Puffin Automation Framework (PAF) query

[Ben McGinnes]

Hello,
I am just wondering if anyone here has managed to get Keyton
Weissinger's Puffin Automation Framework testing application to work?

In particular I'm wrestling with an issue in which a plan will either
execute a set of actions or generate reports, but not both at the same
time.

I'm interested in speaking with anyone who's managed to make this thing
fly or anyone who has a contact point for the author that he actually
responds to.

Feel free to respond either on or off list. Thanks.


Regards,
Ben

 

[Alan Kennedy]

I am just wondering if anyone here has managed to get Keyton
Weissinger's Puffin Automation Framework testing application to work?

I tried it out on a testing job about a year ago. The following are
points of note

1. It is poorly documented. Many important technical details can only
be obtained by reading the source and knowing the behaviour of the
python standard library.

2. Its logging eats memory. Puffin builds an XML DOM to store all of
the results. Which means it eats memory *real* fast.

3. IIRC, It only supports HTTP 1.0, meaning that it does not represent
the actual client->server network activity you would see with real
world apps.

In particular I'm wrestling with an issue in which a plan will either
execute a set of actions or generate reports, but not both at the same
time.

I would only consider using Puffin if you're requirements are very
basic, i.e.

1. You have a fixed set of URL requests that you want to automate.
2. You want to munge data in transmitted and returned resources.
3. You only need to run one of them at a time.

I'm interested in speaking with anyone who's managed to make this thing
fly or anyone who has a contact point for the author that he actually
responds to.

Dunno about contacting Keyton, but feel free to ask questions. I may
be able to answer.

There are some java tools, scriptable in jython, that are much better
all round testing tools, that are more robust. Check this archive
message for details

http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&[email protected]

regards,

 

[Ben McGinnes]

Alan Kennedy([email protected])@Fri, Nov 21, 2003 at 11:19:57AM +0000:

I tried it out on a testing job about a year ago. The following are
points of note

1. It is poorly documented.

Ah, yes, I've discovered that.

Many important technical details can only
be obtained by reading the source and knowing the behaviour of the
python standard library.

That's a bit of a pain, but I've dived into a little Python already,
which is what gave Puffin a good mark on the list.

2. Its logging eats memory. Puffin builds an XML DOM to store all of
the results. Which means it eats memory *real* fast.

That doesn't sound good, but at the moment time is more of an issue than
memory issues.

3. IIRC, It only supports HTTP 1.0, meaning that it does not represent
the actual client->server network activity you would see with real
world apps.

If that's the case it may be a problem, I'll keep my eyes peeled.

I would only consider using Puffin if you're requirements are very
basic, i.e.

1. You have a fixed set of URL requests that you want to automate.

The sort of testing being tried thus far is just a series of simple
simulations (e.g. Joe User logs in and logs out immediately, or logs in
to view a piece of data and logs out). We're not (yet) trying to
actively break the application with invalid data and the like.

2. You want to munge data in transmitted and returned resources.

That may not be much of a concern if we can see a report indicating that
a particular series of actions succeeded or failed.

3. You only need to run one of them at a time.

Given that Puffin can be called from the command line and set with
different "workspaces" (i.e. config files) then it's possible to work
around this. Since the plan was to automate it via cron or similar
anyway this is not a great hassle.

Dunno about contacting Keyton, but feel free to ask questions. I may
be able to answer.

Ah well, it was worth a shot.

The major issue I've encountered at the moment is a conflict between the
plan.xml and action.xml files. From what I can tell the plan.xml file
is supposed to specifically call the actions specified in action.xml in
order to run them. Yet at the moment if puffin detects objects of the
same name in the two files, it breaks with a bunch of Traceback errors.

Ultimately they claim the XML was not well formed, but that XML has been
independently verified as legitimate. The only conflict we can narrow
it down too is the similar names and, indeed, when I change the action
names in plan.xml it appears to work (except that the actions specified
in action.xml are not called, which they need to be).

There are some java tools, scriptable in jython, that are much better
all round testing tools, that are more robust.

I'll certainly have a look, if only for general reference in the future,
but, unfortunately, time still is a primary factor.

Check this archive message for details

http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&[email protected]

Amusingly this thread was initiated by one of the other members of the
project, prior to my joining it. Still, the URLs suggested by yourself
and others in the group will be reviewed once more.


Regards,
Ben

 

[Alan Kennedy]

[Ben McGinnes]

The major issue I've encountered at the moment is a conflict between the
plan.xml and action.xml files. From what I can tell the plan.xml file
is supposed to specifically call the actions specified in action.xml in
order to run them. Yet at the moment if puffin detects objects of the
same name in the two files, it breaks with a bunch of Traceback errors.

Ultimately they claim the XML was not well formed, but that XML has been
independently verified as legitimate. The only conflict we can narrow
it down too is the similar names and, indeed, when I change the action
names in plan.xml it appears to work (except that the actions specified
in action.xml are not called, which they need to be).

What you're discovering is the puffin author's inappropriate use of
XML.

Although the documents you're using may be well-formed, the author has
imposed other implicit constraints on the contents of the XML files
used for configuration. And since these constraints are not
documented, you'll only find them in one of two ways:

1. Tripping over them, as you have just done.
2. Reading the source.

Apart from using a DOM to store the test results (Ugh!), the other
great XML sin committed in Puffin is that old canard: trying to use
XML as a programming language. For example, I just pulled this
fragment out of one of the sample test plans that comes with puffin:

<repeat name="repeat3" count="2">
<task name="makePurchase" dependsList="makeDecision">
<action name="generateSelectionCount"/>
<action name="addItemToCart"/>
<action name="updateCurrentSelection"/>
<action name="showCart"/>
<action name="getUpdatedInventory"/>
</task>
</repeat>

XML as a programming language is a dead-end!

[Alan Kennedy]
[Ben McGinnes]

Amusingly this thread was initiated by one of the other members of the
project, prior to my joining it. Still, the URLs suggested by yourself
and others in the group will be reviewed once more.

I think it would be useful for you to re-read the last message in that
thread, from John J. Lee. Points of note include

1. Webunit may be a better choice than Puffin (I can't say: I haven't
used it).

2. If your requirements are as simple as indicated, then you may find
it far more productive to roll your own using the python standard
library. This should only take a few hours for someone familiar with
the standard library (as your esteemed colleague certainly is .
Alternatively, try a Google Groups search of c.l.py for "automate
httplib" or "automate form".

If time is of the essence, and nobody on the project has the
time/experience for the task, I am available for contract work at
reasonable rates ;-)

regards,

 

[Ben McGinnes]

Alan Kennedy([email protected])@Mon, Nov 24, 2003 at 11:09:00AM +0000:

What you're discovering is the puffin author's inappropriate use of
XML.

Although the documents you're using may be well-formed, the author has
imposed other implicit constraints on the contents of the XML files
used for configuration. And since these constraints are not
documented, you'll only find them in one of two ways:

1. Tripping over them, as you have just done.
2. Reading the source.

Urgh! That is terrible ...

Apart from using a DOM to store the test results (Ugh!), the other
great XML sin committed in Puffin is that old canard: trying to use
XML as a programming language.

Whereas the article on the IBM developerWorks site implied (or perhaps
that should read "impLIEd") that the XML was just used as a method of
parsing configuration parameters.

For example, I just pulled this
fragment out of one of the sample test plans that comes with puffin:

<repeat name="repeat3" count="2">
<task name="makePurchase" dependsList="makeDecision">
<action name="generateSelectionCount"/>
<action name="addItemToCart"/>
<action name="updateCurrentSelection"/>
<action name="showCart"/>
<action name="getUpdatedInventory"/>
</task>
</repeat>

Which is exactly the type of thing I found the conflict with, with the
action name tags above and the actual test actions in the action.xml
file.

XML as a programming language is a dead-end!

I quite agree. I think the program was named after the wrong bird ...
dodo might be more appropriate.

I think it would be useful for you to re-read the last message in that
thread, from John J. Lee. Points of note include

Yeah, that thread has some very useful info in it. Thanks for pointing
it out.

1. Webunit may be a better choice than Puffin (I can't say: I haven't
used it).

I'm looking into WebUnit now (the mechanicalcat.net version, not the
other one mentioned in the thread).

2. If your requirements are as simple as indicated, then you may find
it far more productive to roll your own using the python standard
library. This should only take a few hours for someone familiar with
the standard library (as your esteemed colleague certainly is .
Alternatively, try a Google Groups search of c.l.py for "automate
httplib" or "automate form".

Looks like that's what WebUnit is geared towards, but it may need some
tweaking for our work anyway. Looks like WebUnit's one of the primary
candidates at the moment and small enough to be easily expanded on if
necessary.

If time is of the essence, and nobody on the project has the
time/experience for the task, I am available for contract work at
reasonable rates ;-)

Heh.


Regards,
Ben