Load/Stress/Performance Testing via Ruby

Discussion in 'Ruby' started by Dav Yaginuma, Feb 25, 2009.

  1. Dav Yaginuma

    Dav Yaginuma Guest

    [Note: parts of this message were removed to make it a legal post.]

    I've done some research on this and I'm fairly convinced it doesn't exist,
    but I thought I would just ask here in case I missed something.

    I want to be able to full-stack load test a Ruby on Rails site (not that
    rails matters in this context) running dynamic test plans across multiple
    concurrent users in order to better replicate real world usage.

    In other words, I'd like to run say 50 concurrent user sessions where some
    of them are signing up, some are just browsing around, some are logging in
    and doing UserActivity1, some are doing UserActivity2, etc.

    I want to script the test plans in ruby, similar to a seleinum-rc test plan
    where I can use the same DSL to simulate user input and identify parts of
    the response html, and use that to drive further test activity. I'd like to
    use some of the same DSL from my selenium or integration test helper modules
    already developed in these tests as well in order to be DRY.

    I would like the test framework to mimc the browser behaviour: handling
    cookies and even simulating caching to send the appropriate
    If-Modified-Since headers.

    Since I would like to replicate actual request load on the server, I want to
    make sure that all page assets, even static ones like images/css/js/flash,
    are requested during the test, so it would be nice if an HTTP Proxy could be
    used to create the initial test plans by recording actual browser sessions.

    Some things I've looked at so far:

    JMeter:
    - seems difficult to make dynamic test plans here that will modify requests
    based on earlier responses
    - test plans are in XML, although I suppose I could use a ruby DSL to
    generate the XML, but since the proxy records to XML I'd also have to
    generate ruby from XML

    The Grinder:
    - Looks awesome, and I may end up going with this, but it would be nicer to
    use ruby instead of Jython in order to leverage my teams existing code and
    skills.

    shell/curl/ruby script mish mash:
    - A lot of control, and curl handles cookies, but no real framework support
    and no proxy recording

    Ruby code:
    - Doesn't really seem to be any test framework that deals with concurrent
    requests like this
    - I assume JRuby would be required (and would be sufficient) to get real
    threading
    - I feel like there are probably a lot of pieces that could be tied together
    to make what I want, but unsure of what they are or which ones to
    concentrate on. Hpricot? Mechanize?

    Anyone have any ideas? Is anyone doing load testing via ruby?

    --
    Dav Yaginuma
    http://AkuAku.org/
     
    Dav Yaginuma, Feb 25, 2009
    #1
    1. Advertising

  2. Dav Yaginuma wrote:
    > - I assume JRuby would be required (and would be sufficient) to get real
    > threading


    JRuby or IronRuby would get you native threads, though the latter is
    still pretty alpha.

    I'd love to see a load-testing tool written for real parallel threads in
    Ruby. Maybe something like Grinder?

    - Charlie
     
    Charles Oliver Nutter, Feb 25, 2009
    #2
    1. Advertising

  3. On Wed, Feb 25, 2009 at 11:53 AM, Dav Yaginuma <> wrote:
    > I've done some research on this and I'm fairly convinced it doesn't exist,
    > but I thought I would just ask here in case I missed something.
    >
    > I want to be able to full-stack load test a Ruby on Rails site (not that
    > rails matters in this context) running dynamic test plans across multiple
    > concurrent users in order to better replicate real world usage.
    >
    > In other words, I'd like to run say 50 concurrent user sessions where some
    > of them are signing up, some are just browsing around, some are logging in
    > and doing UserActivity1, some are doing UserActivity2, etc.
    >
    > I want to script the test plans in ruby, similar to a seleinum-rc test plan
    > where I can use the same DSL to simulate user input and identify parts of
    > the response html, and use that to drive further test activity. I'd like to
    > use some of the same DSL from my selenium or integration test helper modules
    > already developed in these tests as well in order to be DRY.
    >
    > I would like the test framework to mimc the browser behaviour: handling
    > cookies and even simulating caching to send the appropriate
    > If-Modified-Since headers.
    >
    > Since I would like to replicate actual request load on the server, I want to
    > make sure that all page assets, even static ones like images/css/js/flash,
    > are requested during the test, so it would be nice if an HTTP Proxy could be
    > used to create the initial test plans by recording actual browser sessions.
    >
    > Some things I've looked at so far:
    >
    > JMeter:
    > - seems difficult to make dynamic test plans here that will modify requests
    > based on earlier responses
    > - test plans are in XML, although I suppose I could use a ruby DSL to
    > generate the XML, but since the proxy records to XML I'd also have to
    > generate ruby from XML
    >
    > The Grinder:
    > - Looks awesome, and I may end up going with this, but it would be nicer to
    > use ruby instead of Jython in order to leverage my teams existing code and
    > skills.
    >
    > shell/curl/ruby script mish mash:
    > - A lot of control, and curl handles cookies, but no real framework support
    > and no proxy recording
    >
    > Ruby code:
    > - Doesn't really seem to be any test framework that deals with concurrent
    > requests like this
    > - I assume JRuby would be required (and would be sufficient) to get real
    > threading
    > - I feel like there are probably a lot of pieces that could be tied together
    > to make what I want, but unsure of what they are or which ones to
    > concentrate on. Hpricot? Mechanize?
    >
    > Anyone have any ideas? Is anyone doing load testing via ruby?
    >
    > --
    > Dav Yaginuma
    > http://AkuAku.org/
    >


    I've made similar searches and come up with the same emptiness. There
    are two major *commercial* load testing tools, LoadRunner and
    SilkPerformer. I have about seven years of experience with
    SilkPerformer and almost none with LoadRunner. There are a few open
    source recorders, but none I have found that are anywhere near as good
    as SilkPerformer.

    What you *could* do is get a low-end SilkPerformer or LoadRunner
    license ... just enough virtual users so you'd get a recording
    capability ... and then write a Ruby run-time and parser to translate
    the "native" language to Ruby. But check the EULA to see if this is
    legal. I never tried it because I consider it a waste of effort.

    Really, you need to make a business decision here. What is the cost to
    your business of the time you spend building a load testing framework
    when you could be developing and testing your product? I'd recommend
    you seriously consider buying a license for LoadRunner or
    SilkPerformer if performance of your application is mission-critical.

    --
    M. Edward (Ed) Borasky
    http://www.linkedin.com/in/edborasky

    I've never met a happy clam. In fact, most of them were pretty steamed.
     
    M. Edward (Ed) Borasky, Feb 25, 2009
    #3
  4. Dav Yaginuma

    Dav Yaginuma Guest

    [Note: parts of this message were removed to make it a legal post.]

    On Wed, Feb 25, 2009 at 12:58 PM, M. Edward (Ed) Borasky
    <>wrote:

    > On Wed, Feb 25, 2009 at 11:53 AM, Dav Yaginuma <>
    > wrote:
    > > I've done some research on this and I'm fairly convinced it doesn't

    > exist,
    > > but I thought I would just ask here in case I missed something.
    > >
    > > I want to be able to full-stack load test a Ruby on Rails site (not that
    > > rails matters in this context) running dynamic test plans across multiple
    > > concurrent users in order to better replicate real world usage.
    > >
    > > In other words, I'd like to run say 50 concurrent user sessions where

    > some
    > > of them are signing up, some are just browsing around, some are logging

    > in
    > > and doing UserActivity1, some are doing UserActivity2, etc.
    > >
    > > I want to script the test plans in ruby, similar to a seleinum-rc test

    > plan
    > > where I can use the same DSL to simulate user input and identify parts of
    > > the response html, and use that to drive further test activity. I'd like

    > to
    > > use some of the same DSL from my selenium or integration test helper

    > modules
    > > already developed in these tests as well in order to be DRY.
    > >
    > > I would like the test framework to mimc the browser behaviour: handling
    > > cookies and even simulating caching to send the appropriate
    > > If-Modified-Since headers.
    > >
    > > Since I would like to replicate actual request load on the server, I want

    > to
    > > make sure that all page assets, even static ones like

    > images/css/js/flash,
    > > are requested during the test, so it would be nice if an HTTP Proxy could

    > be
    > > used to create the initial test plans by recording actual browser

    > sessions.
    > >
    > > Some things I've looked at so far:
    > >
    > > JMeter:
    > > - seems difficult to make dynamic test plans here that will modify

    > requests
    > > based on earlier responses
    > > - test plans are in XML, although I suppose I could use a ruby DSL to
    > > generate the XML, but since the proxy records to XML I'd also have to
    > > generate ruby from XML
    > >
    > > The Grinder:
    > > - Looks awesome, and I may end up going with this, but it would be nicer

    > to
    > > use ruby instead of Jython in order to leverage my teams existing code

    > and
    > > skills.
    > >
    > > shell/curl/ruby script mish mash:
    > > - A lot of control, and curl handles cookies, but no real framework

    > support
    > > and no proxy recording
    > >
    > > Ruby code:
    > > - Doesn't really seem to be any test framework that deals with concurrent
    > > requests like this
    > > - I assume JRuby would be required (and would be sufficient) to get real
    > > threading
    > > - I feel like there are probably a lot of pieces that could be tied

    > together
    > > to make what I want, but unsure of what they are or which ones to
    > > concentrate on. Hpricot? Mechanize?
    > >
    > > Anyone have any ideas? Is anyone doing load testing via ruby?
    > >
    > > --
    > > Dav Yaginuma
    > > http://AkuAku.org/
    > >

    >
    > I've made similar searches and come up with the same emptiness. There
    > are two major *commercial* load testing tools, LoadRunner and
    > SilkPerformer. I have about seven years of experience with
    > SilkPerformer and almost none with LoadRunner. There are a few open
    > source recorders, but none I have found that are anywhere near as good
    > as SilkPerformer.
    >
    > What you *could* do is get a low-end SilkPerformer or LoadRunner
    > license ... just enough virtual users so you'd get a recording
    > capability ... and then write a Ruby run-time and parser to translate
    > the "native" language to Ruby. But check the EULA to see if this is
    > legal. I never tried it because I consider it a waste of effort.
    >
    > Really, you need to make a business decision here. What is the cost to
    > your business of the time you spend building a load testing framework
    > when you could be developing and testing your product? I'd recommend
    > you seriously consider buying a license for LoadRunner or
    > SilkPerformer if performance of your application is mission-critical.
    >
    > --
    > M. Edward (Ed) Borasky
    > http://www.linkedin.com/in/edborasky
    >
    > I've never met a happy clam. In fact, most of them were pretty steamed.
    >
    >



    Thanks Ed,

    Good to know I wasn't missing something obvious.

    I've contacted sales reps for SilkPerformer and LoadRunner. My fear (and
    unfortunate experience) with this kind of software is that you spend a lot
    to get a complicated but powerful system that then requires more time (and
    money) to make actually work. They both seem to have a lot of features that
    I don't need, so I was hoping to find something more streamlined and
    ruby-oriented first. It's good to be aware of all the alternatives though,
    so thanks for the pointer. I had heard of LoadRunner but not SilkPerformer.


    --
    Dav Yaginuma
    http://AkuAku.org/
     
    Dav Yaginuma, Feb 26, 2009
    #4
  5. Dav Yaginuma wrote:
    > Thanks Ed,
    >
    > Good to know I wasn't missing something obvious.
    >
    > I've contacted sales reps for SilkPerformer and LoadRunner. My fear (and
    > unfortunate experience) with this kind of software is that you spend a lot
    > to get a complicated but powerful system that then requires more time (and
    > money) to make actually work. They both seem to have a lot of features that
    > I don't need, so I was hoping to find something more streamlined and
    > ruby-oriented first. It's good to be aware of all the alternatives though,
    > so thanks for the pointer. I had heard of LoadRunner but not SilkPerformer.
    >
    >

    50 concurrent virtual users probably won't cost you a lot. They usually
    charge a base price plus a price per virtual user. The assumption is
    that your business will grow, you'll need to emulate more users, and
    you'll be able to buy more virtual users.

    For what it's worth, the real "secret sauce" in the recorders is the
    ability to extract session data, cookie send / receive, parsing data out
    of the returned HTML and branching on it, etc. Pure record / playback
    isn't all that hard to do, it's the "other stuff" that makes these tools
    worth the money.

    P.S.: It warms my heart to know that there are actually web application
    developers out there who realize the *need* for load testing. There are
    some big name web sites out there that seem to go into deep space when
    they update the database, for example. And load testing can also give
    you *good* news ... you may need *less* server than you're planning on.

    --
    M. Edward (Ed) Borasky
    http://www.linkedin.com/in/edborasky

    I've never met a happy clam. In fact, most of them were pretty steamed.
     
    M. Edward (Ed) Borasky, Feb 26, 2009
    #5
  6. I have great experiences with tsung:

    http://tsung.erlang-projects.org/

    Its easy to distribute, rather easy to configure (in XML...), is able
    to handle dynamic stuff like "create a resource, parse the id out of
    the answer and create a child resource" rather easily. There is also
    an easy way of consuming the answer using an Erlang function.[1] The
    Documentation is quite thorough.

    It also provides a recording proxy and AFAIK there is a parser for
    apache logs.

    I also have a small sinatra-app serving the logs that I want to
    release next week or so.

    Regards,
    Florian

    [1]: It's not that hard.

    --
    Florian Gilcher

    smtp:
    jabber:
    gpg: 533148E2
     
    Florian Gilcher, Feb 26, 2009
    #6
  7. Dav Yaginuma

    Kyle Schmitt Guest

    On Wed, Feb 25, 2009 at 1:53 PM, Dav Yaginuma <> wrote:
    > JMeter:
    > - seems difficult to make dynamic test plans here that will modify requests
    > based on earlier responses
    > - test plans are in XML, although I suppose I could use a ruby DSL to
    > generate the XML, but since the proxy records to XML I'd also have to
    > generate ruby from XML


    At first it looks rather difficult to make dynamic tests in JMeter,
    but it's really not hard once you get into it. The caveat is, you use
    javascript & regexes & xml or CSV files. I used to do this frequently
    at my old job.

    Under no circumstances even bother with TestComplete by AutomatedQA.
    It's a piece of junk.

    Unless you have the cash to spend on expensive software and a few
    training classes for it, I would delve further into JMeter. Record
    snippets of what you want to do into multiple test groups, trim down
    the calls to just what you need, code from there. You may want to
    start following their mailing list. It helped me a ton.

    --Kyle
     
    Kyle Schmitt, Feb 26, 2009
    #7
  8. Dav Yaginuma

    Peter Booth Guest

    [Note: parts of this message were removed to make it a legal post.]

    Dav,

    It really depends on why you want to do this. If you simply want to
    push a workload that is representative but without any concern about
    absolute correctness then a JMeter or LoadRunner or Grinder can be
    good. LoadRunner has a great UI. This allows you to stress test and
    see where the hotspots are. New Relic is an excellent complement to
    any of these.

    If you want a realistic load test and want to actually estimate
    absolute capacity then you're choosing to do something harder. The
    best tool is the open source httperf. Its the only tool here that uses
    an open workload model. You will find that LoadRunner and the other
    tools I suggested will always overestimate system capacity. That is
    because in these tools, a virtual agent won't fire a new user request
    until the prior one completes.

    This of course is not what happens in real life, where a heavy tailed
    Pareto distribution of requests is typical. Httperf addresses this
    issue and allows you to get data that is more realistic in an absolute
    sense.

    If you're wanting to tune and improve performance or throughput and
    not simply measure it that's a whole other ball-game. Be prepared to
    take some time and learn, or hire some help.

    Peter Booth
    917 445 5663

    PS - If anyone needs assistance with tuning Rails apps, website
    capacity planning, load testing, please ping me. I am available. I
    work at the application code, web server, DB server, operating
    system, TCP stack, physical network and hardware layers.

    On Wed, Feb 25, 2009 at 11:53 AM, Dav Yaginuma <>
    wrote:
    > I've done some research on this and I'm fairly convinced it doesn't
    > exist,
    > but I thought I would just ask here in case I missed something.
    >
    > I want to be able to full-stack load test a Ruby on Rails site (not
    > that
    > rails matters in this context) running dynamic test plans across
    > multiple
    > concurrent users in order to better replicate real world usage.
    >
    > In other words, I'd like to run say 50 concurrent user sessions
    > where some
    > of them are signing up, some are just browsing around, some are
    > logging in
    > and doing UserActivity1, some are doing UserActivity2, etc.
    >
    > I want to script the test plans in ruby, similar to a seleinum-rc
    > test plan
    > where I can use the same DSL to simulate user input and identify
    > parts of
    > the response html, and use that to drive further test activity. I'd
    > like to
    > use some of the same DSL from my selenium or integration test helper
    > modules
    > already developed in these tests as well in order to be DRY.
    >
    > I would like the test framework to mimc the browser behaviour:
    > handling
    > cookies and even simulating caching to send the appropriate
    > If-Modified-Since headers.
    >
    > Since I would like to replicate actual request load on the server, I
    > want to
    > make sure that all page assets, even static ones like images/css/js/
    > flash,
    > are requested during the test, so it would be nice if an HTTP Proxy
    > could be
    > used to create the initial test plans by recording actual browser
    > sessions.
    >
    > Some things I've looked at so far:
    >
    > JMeter:
    > - seems difficult to make dynamic test plans here that will modify
    > requests
    > based on earlier responses
    > - test plans are in XML, although I suppose I could use a ruby DSL to
    > generate the XML, but since the proxy records to XML I'd also have to
    > generate ruby from XML
    >
    > The Grinder:
    > - Looks awesome, and I may end up going with this, but it would be
    > nicer to
    > use ruby instead of Jython in order to leverage my teams existing
    > code and
    > skills.
    >
    > shell/curl/ruby script mish mash:
    > - A lot of control, and curl handles cookies, but no real framework
    > support
    > and no proxy recording
    >
    > Ruby code:
    > - Doesn't really seem to be any test framework that deals with
    > concurrent
    > requests like this
    > - I assume JRuby would be required (and would be sufficient) to get
    > real
    > threading
    > - I feel like there are probably a lot of pieces that could be tied
    > together
    > to make what I want, but unsure of what they are or which ones to
    > concentrate on. Hpricot? Mechanize?
    >
    > Anyone have any ideas? Is anyone doing load testing via ruby?
    >
    > --
    > Dav Yaginuma
    > http://AkuAku.org/
    >
     
    Peter Booth, Feb 26, 2009
    #8
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?bWlrZQ==?=

    web app stress testing

    =?Utf-8?B?bWlrZQ==?=, Aug 17, 2004, in forum: ASP .Net
    Replies:
    0
    Views:
    386
    =?Utf-8?B?bWlrZQ==?=
    Aug 17, 2004
  2. cpnet

    Stress testing help

    cpnet, Sep 23, 2004, in forum: ASP .Net
    Replies:
    7
    Views:
    503
    Steven Cheng[MSFT]
    Sep 27, 2004
  3. =?Utf-8?B?R2xlbm4=?=

    Stress Testing Using Forms Authentication

    =?Utf-8?B?R2xlbm4=?=, Oct 29, 2004, in forum: ASP .Net
    Replies:
    1
    Views:
    1,246
    Scott Allen
    Oct 31, 2004
  4. Ravi Chand via .NET 247

    Errors while stress testing a load balanced application

    Ravi Chand via .NET 247, May 20, 2005, in forum: ASP .Net
    Replies:
    0
    Views:
    494
    Ravi Chand via .NET 247
    May 20, 2005
  5. =?Utf-8?B?Um9iZXJ0?=

    Stress Testing

    =?Utf-8?B?Um9iZXJ0?=, Jun 19, 2006, in forum: ASP .Net
    Replies:
    1
    Views:
    414
    Kevin Spencer
    Jun 20, 2006
Loading...

Share This Page