Ruby replacement for ab?

M

Mat Schaffer

So I've been hunting for a way to stress test a web application and
haven't been terribly satisfied so far.

I'd really love something where I could define a script that
determined the way an agent would behave. Basically a set of
requests where each request had the potential to use information
contained in the return of a previous request.

My brain is slowly churning around the possibilities of putting
ruby's benchmark.rb, mechanize, maybe drb to manage multiple test
hosts.... Some sort of ruby DSL would be sweet. But before I get
too ahead of myself: Does anyone here know of a tool does this?

I'm looking at JMeter right now. It at least allows for assertions
on HTTP requests as part of an agent script, but I don't see any
facility to do computations on the returned content from a previous
request.

Thoughts? Suggestions?

Thanks in advance,
Mat
 
H

hemant

So I've been hunting for a way to stress test a web application and
haven't been terribly satisfied so far.

I'd really love something where I could define a script that
determined the way an agent would behave. Basically a set of
requests where each request had the potential to use information
contained in the return of a previous request.

My brain is slowly churning around the possibilities of putting
ruby's benchmark.rb, mechanize, maybe drb to manage multiple test
hosts.... Some sort of ruby DSL would be sweet. But before I get
too ahead of myself: Does anyone here know of a tool does this?

I'm looking at JMeter right now. It at least allows for assertions
on HTTP requests as part of an agent script, but I don't see any
facility to do computations on the returned content from a previous
request.

Thoughts? Suggestions?

Thanks in advance,
Mat

Did you see, : http://rfuzz.rubyforge.org/
 
K

khaines


rfuzz is nice but isn't really what he's asking about.

What he is asking about is something that I have wanted for a long time.
I want to be able to quickly and easily setup a test script for a site.

Go to a URL. Follow a link on the page returned. Maybe pick randomly
from a few choices of links to follow. Repeat a few times to simulate
people clicking around a site. Go to a page with a form, and fill in form
fields, etc...

Basically, a DSL to write scripts for driving interactions to a site or
application that would somewhat simulate what real traffic is expected to
look like.

That would be dandy.


Kirk Haines
 
F

Fred Wulff

It seems like it would be fairly easy to set up something with
WWW::Mechanize, although there might be performance problems if you
wanted to use it for stress testing.

-Fred
 
K

khaines

It seems like it would be fairly easy to set up something with
WWW::Mechanize, although there might be performance problems if you
wanted to use it for stress testing.

Yeah, possibly. Mechanize or hpricot could be part of a solution, but the
real challenge is in devising a DSL that makes it _easy_ for me to, for
example:

1) Hit the homepage.

2) Go to a random URL on the site.
Repeat 2-5 times.

3) Go to the fund information page.

4 Click through to the distributions display.

5) Randomly select 1-4 fund information links.

Repeat 100000 times, collecting timing information, failure information,
etc....



Then, if I can run that on two or three machines at the same time, and can
collate their logs and run an analysis on the collated logs, I'd be a
happy fellow.

Another thing that I'd love to be able to do is to run something like the
above set of actions, but do it more slowly, with pauses between each
request, so that the site may only get, on average, one request a second
but I can let it run for days.

I have encountered failures on occasion that didn't seem to happen as a
result of short term heavy loading, but more as a result of steady hits
over a longer period of time.

I'd like to be able to easily simulate that usage pattern in a development
environment.


Kirk Haines
 
P

pat eyler

So I've been hunting for a way to stress test a web application and
haven't been terribly satisfied so far.

I'd really love something where I could define a script that
determined the way an agent would behave. Basically a set of
requests where each request had the potential to use information
contained in the return of a previous request.

My brain is slowly churning around the possibilities of putting
ruby's benchmark.rb, mechanize, maybe drb to manage multiple test
hosts.... Some sort of ruby DSL would be sweet. But before I get
too ahead of myself: Does anyone here know of a tool does this?

I'm looking at JMeter right now. It at least allows for assertions
on HTTP requests as part of an agent script, but I don't see any
facility to do computations on the returned content from a previous
request.

Thoughts? Suggestions?

I'm not really maintaining it at this point, but RWB might help. it's
available as a gem. I've depracated it in favor of RFuzz, but RFuzz
still hasn't caught all the features.
 
M

Mat Schaffer

I'm not really maintaining it at this point, but RWB might help. it's
available as a gem. I've depracated it in favor of RFuzz, but RFuzz
still hasn't caught all the features.

Both RFuzz and RWB look like really good starts toward what I'm after.
Tough to say if I'll decide to really code something at this point.
But I'll certainly start browsing the source of both projects to get
some ideas.

I'd love to have something that made it easy to take a script (like in
rfuzz or rwb), then just drop it straight into a harness that was
responsible for multi-threading, distribution across slave nodes,
aggregating data and generating reports.

But given the state of things, it'd be a big problem to tackle.
Possibly too big for me at this point.

Thanks for the feedback. This list never disappoints!
-Mat
 
P

pat eyler

Both RFuzz and RWB look like really good starts toward what I'm after.
Tough to say if I'll decide to really code something at this point.
But I'll certainly start browsing the source of both projects to get
some ideas.

If you're planning on writing something, either coordinate with Zed
to work on RFuzz, or take over RWB and go for it.

I'd love to have something that made it easy to take a script (like in
rfuzz or rwb), then just drop it straight into a harness that was
responsible for multi-threading, distribution across slave nodes,
aggregating data and generating reports.

That's sort of where I was headed with RWB. It's already multithreaded,
but it really should use dRb to coordinate across multiple nodes. I
don't think there's too much code needed to do that.
But given the state of things, it'd be a big problem to tackle.
Possibly too big for me at this point.

If you decide to go with RWB, I'd be happy to answer any questions.
It might be a plateful, but you can take small bites and it won't be
that hard. (Same is true of RFuzz, except that I can't answer the
questions.)
 
X

Xavier Noria

Go to a URL. Follow a link on the page returned. Maybe pick
randomly from a few choices of links to follow. Repeat a few times
to simulate people clicking around a site. Go to a page with a
form, and fill in form fields, etc...

Basically, a DSL to write scripts for driving interactions to a
site or application that would somewhat simulate what real traffic
is expected to look like.

Do you know Watir?

-- fxn
 
M

Mat Schaffer

Um, then what DO you need that Rails unit and integration testing can't
handle? Any other type of testing *requires* a browser instance.

Jason

I can't answer Kirk's question, but personally I'm looking for full
stack stress testing. Mostly so I can give my superiors some evidence
rather than blinding estimating how many concurrent users we can
handle. Not exactly mission-critical, but warm fuzzies are always
good.
-Mat
 
K

khaines

See below. Rails unit testing is irrelevant. I don't use Rails. It's
inadequate for my needs.
I can't answer Kirk's question, but personally I'm looking for full
stack stress testing. Mostly so I can give my superiors some evidence
rather than blinding estimating how many concurrent users we can
handle. Not exactly mission-critical, but warm fuzzies are always
good.

What I would like, and what Mat wants, I think, have a great deal of
overlap.

To repeat some of what I said in an earlier post, I'd like to.

1) Hit the homepage of a site.
2) Go to a random URL on the site.
Repeat 2-5 times.
3) Go to the fund information page.
4 Click through to the distributions display.
5) Randomly select 1-4 fund information links.

Repeat 100000 times, collecting timing information, failure information,
etc....

Or, even more ambitiously:

1) Hit the homepage of a site.
2) Follow the link to the advisor login page.
3) Login (an action that requires sending a form response, and receiving a
cookie in response that will be sent on further requests).
4) View fund information for some random funds.
5) Go into advisor tools and feed some numbers into a calculator.
6) Goto the logout link.


If I can create scripts that define actions like those outlined above,
which would can simulate expected real world usage over a period of days,
or can simulate hordes of users in a more realistic manner than the
ab/httperf mechanism of hammering a single URL, it would be very valuable.

I don't need to drive an actual IE or Firefox session, and for this sort
of thing that's impractical.


Thanks,

Kirk Haines
 
M

M. Edward (Ed) Borasky

What I would like, and what Mat wants, I think, have a great deal of
overlap.

To repeat some of what I said in an earlier post, I'd like to.

1) Hit the homepage of a site.
2) Go to a random URL on the site.
Repeat 2-5 times.
3) Go to the fund information page.
4 Click through to the distributions display.
5) Randomly select 1-4 fund information links.

Repeat 100000 times, collecting timing information, failure
information, etc....

Or, even more ambitiously:

1) Hit the homepage of a site.
2) Follow the link to the advisor login page.
3) Login (an action that requires sending a form response, and
receiving a cookie in response that will be sent on further requests).
4) View fund information for some random funds.
5) Go into advisor tools and feed some numbers into a calculator.
6) Goto the logout link.


If I can create scripts that define actions like those outlined above,
which would can simulate expected real world usage over a period of
days, or can simulate hordes of users in a more realistic manner than
the ab/httperf mechanism of hammering a single URL, it would be very
valuable.

I don't need to drive an actual IE or Firefox session, and for this
sort of thing that's impractical.
What you are describing is a load testing tool. I work with one such
tool, Segue (now Borland) SilkPerformer. Another well-known one is
Mercury LoadRunner. As far as I know, there are no open source tools
that have the flexibility in script *capture* these tools have.
Essentially, you fire up a script recorder, it fires up a browser, you
exercise your app, you close the recorder and it generates a script.
There are also tools for load test monitoring, server monitoring and
performance/bottleneck analysis. These things are expensive, though.

The closest thing I've found in the open source world is jMeter. But as
far as I know, you have to code your own scripts, your own benchmarks
and your own performance analysis tools with jMeter. It's a tradeoff of
money versus effort. Quite frankly, I'd recommend spending the money.
Email me off-line if you want the details.

Now if someone wants to *write* a full-featured web application load
testing tool -- and I mean *full-featured* -- in Ruby and open source
it, allow me to encourage them. I sure as heck don't have the time to do
it. :)
 
H

Hassan Schroeder

.... As far as I know, there are no open source tools
that have the flexibility in script *capture* these tools have.
Essentially, you fire up a script recorder, it fires up a browser, you
exercise your app, you close the recorder and it generates a script.

Perhaps you missed my prior post about WebTest and associated
WebTestRecorder? :) Yes, it's Java, but if you're considering JMeter...

Caveat: I haven't used either of the commercial tools you mentioned,
so grain of salt, YMMV, etc.
 
M

M. Edward (Ed) Borasky

Hassan said:
Perhaps you missed my prior post about WebTest and associated
WebTestRecorder? :) Yes, it's Java, but if you're considering JMeter...

Caveat: I haven't used either of the commercial tools you mentioned,
so grain of salt, YMMV, etc.
Yeah ... I missed it when I first posted but I have just looked at it.
Unfortunately, the recorder apparently only works with Firefox, which is
a deal-killer. However much you may hate IE and Microsoft ...
 
M

M. Edward (Ed) Borasky

Jason said:
Could you explain why this is a "deal killer"? I would hope that
per-browser
functionality testing is a seperate suite from load testing, unless
you're
sending different websites depending on the browser type.

Jason
If the application only works with Internet Explorer, I can't record it
with Firefox.
 
K

khaines

Perhaps you missed my prior post about WebTest and associated
WebTestRecorder? :) Yes, it's Java, but if you're considering JMeter...

WebTest, from my cursory look, seems to be more oriented at validating the
correctness of a web site/application than stress testing, though. Did I
miss something?


Kirk Haines
 
H

Hassan Schroeder

WebTest, from my cursory look, seems to be more oriented at validating the
correctness of a web site/application than stress testing, though. Did I
miss something?

If you want more complex interactions than JMeter provides, there's
no reason you can't create WebTest scripts and set off multiple copies
running simultaneously. But no, it's not intrinsically designed for stress
testing.
 
M

M. Edward (Ed) Borasky

Hassan said:
If you want more complex interactions than JMeter provides, there's
no reason you can't create WebTest scripts and set off multiple copies
running simultaneously. But no, it's not intrinsically designed for
stress
testing.
OK ... let's get our definitions straight.

1. QA/Correctness testing: if I enter request A, does the web app return
the correct response? If I buy a gizmo, does the database show one less
gizmo available to the next buyer? Can Joe Visitor log in and read
Francine User's credit card numbers?

2. Stress test: how gracelessly does the web app behave under an
attempted DoS attack?

3. Load and scalability testing: Assuming it returns the correct
responses, what kind of realistic user load can the web app and its
servers sustain? Where are the bottlenecks? Are they software or
hardware? If software, how can I fix them? How can I inform my designers
how to do things better the next time? If it's hardware, what kind of
hardware? Network? Disk? Server?

Number 3 presupposes you've done Number 1! In fact, the load test
scripts must, to be valid, check the data coming back from the
application to make sure the response is correct. In any event,
SilkPerformer and LoadRunner are specifically designed and evolved to do
3. I didn't dig into Web Test to see if it could be made to do 3 or how
much effort would be involved, since it doesn't record IE. Considering
how long SilkPerformer and LoadRunner have been around, I rather doubt
that Web Test has the kind of "industrial strength" load and scalability
testing capabilities.

I realize I sound like a salesman for a couple of rather expensive tools
-- if that offends the denizens of this open source list, well, I'm
sorry. As I said before, if somebody wants to go out and build an open
source tool in Ruby that does industrial strength load and scalability
testing, I'll be very encouraging, at least with "moral support". But
it's a big job -- it's not as simple as it looks. It's certainly more
than I'd be willing to take on as a one-man open source hacker.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top