Example of web application done right?

G

Glen Pfeiffer

Hi all. I am building a new web application and am evaluating different
technologies. I was wondering if anyone can recommend a large web
application in Perl that was "done right" so I can look at the source
code to evaluate it and determine if I want to use Perl.

This is not a "What is the best language?" question. I merely want to
look at an application that is easy to extend/modify/maintain which
separates logic from display code, and compare it to some other
applications done in other languages. I have experience building large
sites in other languages, but not in Perl. Some thoughts I have had are
WebGUI, YaBB and YAWP.

Thanks,
Glen
 
C

Chris

Glen said:
Hi all. I am building a new web application and am evaluating different
technologies. I was wondering if anyone can recommend a large web
application in Perl that was "done right" so I can look at the source
code to evaluate it and determine if I want to use Perl.

This is not a "What is the best language?" question. I merely want to
look at an application that is easy to extend/modify/maintain which
separates logic from display code, and compare it to some other
applications done in other languages. I have experience building large
sites in other languages, but not in Perl. Some thoughts I have had are
WebGUI, YaBB and YAWP.

Moveable Type. It's written entirely in Perl, makes excellent use of
Perl OOP and modules and uses my personal favorite web templating system
-- HTML::Template. You can download the full source from
http://www.moveabletype.org.

I would consider any web site written in Perl that made use of
HTML::Template or Template Toolkit to be at least a step in the right
direction and for the author of such to have some understanding of
separating logic from presentation layer which I consider absolutely
necessary. I seldom write *test* code in Perl apart from HTML::Template
for that very reason. I would think you would definitely want to look
at those two tools.

There was also very recently -- within the last year -- a sample web
application in Perl written up in either _Linux Magazine_ or _Linux
Journal_ that made use of Template Toolkit that looked pretty good too.
If I have a chance to look it up later I will and report here.

Randall Schwartz uses Template Toolkit also so you might check out his
StoneHenge site: http://www.stonehenge.com

Chris
 
T

thumb_42

Glen Pfeiffer said:
Hi all. I am building a new web application and am evaluating different
technologies. I was wondering if anyone can recommend a large web
application in Perl that was "done right" so I can look at the source
code to evaluate it and determine if I want to use Perl.

This is not a "What is the best language?" question. I merely want to
look at an application that is easy to extend/modify/maintain which
separates logic from display code, and compare it to some other
applications done in other languages. I have experience building large
sites in other languages, but not in Perl. Some thoughts I have had are
WebGUI, YaBB and YAWP.

I can't give you the code, as it was proprietary :-( But I can explain the
design:

We used an 'N'-tier model, with perl daemon(s) running on several machines.
The web portion used a distributed directory to look for a list of back-ends
that were currently available, contacted the backend, submitted the
operation(s) and processed the results. (here operations would refer to
database inserts, user add, queries, fetching mirrored file data, etc..)

The idea was that these back-ends would do all the heavy lifting, provide
additional scale (need more processing? add more machines) and completely
abstract the database from the web front.

The back-end contacted a database for a great deal of it's work. To
accomplish portability (and improve maintenance) we used an XML format for
each SQL statement the system used. (this was of course, cached. we didn't
parse the document every time we needed a database operation) this XML
document of SQL statements proved very useful, we were able to port
everything from postgresql to DB2 in about 45 minutes. :) It also proved
extremely useful for tuning later on as the database grew in size, the
SQL was clearly visable and easily modified by altering this one document.

Keeping database handles around on the web side proved to be very
problematic, even with mod_perl. I would not recommend it. We ended up
ripping every bit of code that referred to a database handle out of the web
front.

We were successfully able to take down any machine for maintenance without a
service outage, with the exception of the database server. (and we did take
down the servers 1 by one on a few occassions w/out service outages)

This was a very important feature for this project, because as you know,
servers need upgrades and maintenance. Having redundancy is important, but
everything has to be designed to work in that type of environment.

In theory, we would have been able to take down the database server too
if we had the license and hardware to mirror it, and/or had designed some
of these back-ends to "queue" data inserts.

As far as the web front.. I would really not recommend perl for this
portion. (at this time) But, if you're going to do it, keep it a very simple
model-view-controller, and use a templating system that makes sense for your
needs. Be very wary of any session handling, as soon as you need session
data you'll have a mess in keeping synchronized sessions across multiple web
hosts. (session handling was by far the biggest issue we had with this
thing)

Jamie
 
I

Ian Cass

Glen said:
This is not a "What is the best language?" question. I merely want to
look at an application that is easy to extend/modify/maintain which
separates logic from display code, and compare it to some other
applications done in other languages. I have experience building large
sites in other languages, but not in Perl. Some thoughts I have had
are WebGUI, YaBB and YAWP.

I always use Apache::ASP for my web stuff. www.apache-asp.org See their
Testimonials section for examples of large websites using Apache::ASP.
 
S

Simon McCaughey

As far as the web front.. I would really not recommend perl for this

What would you recommend? and what did you use?
portion. (at this time) But, if you're going to do it, keep it a very simple
model-view-controller, and use a templating system that makes sense for your
needs. Be very wary of any session handling, as soon as you need session
data you'll have a mess in keeping synchronized sessions across multiple web
hosts. (session handling was by far the biggest issue we had with this
thing)

Did you manage without session data? or how did you handle this problem?

Simon
 
T

thumb_42

Simon McCaughey said:
What would you recommend? and what did you use?

We used perl, and regretted it for the web interface.

I might go with java servlets if I were to do it again, or try and find a
way to get PHP to do it. mod_perl wasn't very stable IMO.
Did you manage without session data? or how did you handle this problem?

No.. the sessions were a constant struggle. Here is how it was supposed to
work (as I recall):

Session ID was a string that contained info about which machine it was on.

If Session ID is from another machine, fetch session across network
and make it local, use new session ID.

Our web server load balancing was done by DNS, and, thankfully most browsers
didn't rotate the IP, so we didn't have to copy the session across very often,
when we did, it usually worked but on occassion it dropped them on the
floor. :-(

I much perfer the way Apache + tomcat handle the problem, even if you use
perl to do the web interface, it's a technique that is worth looking at.

Basically the way they do it is the web server is like a proxy that forwards
the HTTP request to whoever is responsible for that that session ID. I
suppose we could have used this approach in our code as well but didn't
think of it then. :)

It's kind of a matter of cost -vs- cost of failure type of problem. It's a
hassle, but not a complete outage because (ideally) the (current_users /
(number of servers - 1) ) are unaffected by it, and anyway, affected users
could restart their session. Considering most of your "heavy lifting" is
done on a backend server, taking a web server down is relatively rare anyway
so this isn't that huge of a concern. If you were a stock broker it might be
though. (in that case you should probably have dedicated servers that do
nothing but manage sessions)

The most important part of doing large-scale web applications (IMO) is
preparing for the maintenance issues, second is hardware failures. Ideally,
you can work it out so that any machine can be taken out with minimal
or no impact.

A big part of that was dynamic configuration, when you take a machine out or
add a new one in, all the other machines need to know about it. We "abused"
an LDAP server for this purpose. Using LDAP was overkill, but we could take
a backend out of the loop, wait for it to finish what it was doing then stop
it, perform an upgrade, start it, and tell the other machines about it
via LDAP. (repeat process foreach backend until everything is updated) This
was actually kind of cool because it meant we didn't have to login to each
machine to update it's config. :)

By keeping as much logic as possible outside of the web space and on
back-end(s), most maintenance was handled 100% transparently.

Jamie
 
C

Chris

We used perl, and regretted it for the web interface.

What exactly does that mean? If by this you mean that you used 'print'
statements to your HTTPD output stream (eg. print qq{<table border=0
cellpadding=0 cellspacing=0>}) then, yes, I would agree with you. On
that count NO language that outputs to an HTTPD stdout stream gets along
very well, Perl "here" documents are an improvement and better most
languages, but still aren't the answer in large-scale web applications.

If by "we used Perl" you mean you used something like HTML::Template or
Template Toolkit, then I am surprised at your statement. IMO,
HTML::Template coupled with Perl, separates the logic from the
presentation about as well as anything and provides much flexibilty and
power. I much prefer this technique to PHP. It sounds like you didn't
use a templating solution.

I'm not trying to sound bigoted because this is c.l.p.m. I'm *in*
c.l.p.m because I've *been* bigoted *by* Perl. (I hope you see the
difference). I've used a lot of so called "web solutions" out there --
ASP, PHP, JSP, CF, Ruby, even .NET etc. -- and I keep coming back to
Perl. It rocks them all AFAIC.
I might go with java servlets if I were to do it again, or try and find a
way to get PHP to do it. mod_perl wasn't very stable IMO.

I'm very surprised at this statement as well; mod_perl is "the bomb" --
it's rock solid. mod_perl on an Apache server can do ANYTHING. I'm not
trying to cast a shadow on you or the resources you had available to
you, but I can't help but express surprise at some of your statements,
because entirely the opposite has been true for me.

Give me Perl/mod_perl, Apache, MySQL, and HTML::Template and I'll put
together a multi-tiered, multi-server, load-balanced web solution that
would better anything (probably faster) put together by Java, PHP or
JSP. My opinion, but I'll bet many here would second it.
No.. the sessions were a constant struggle.

The most pain-free solution I have come up with for "sessions" in a
load-balancing situation is to take this data to the back-end or add
this as a separate tier as well. The extra time and resources taken for
lookups is worth the pain it removes esp. if your back-end solution is
tuned properly.

I'm hoping soon to see the Berkeley XML DB put through it's paces in
just such a scenario. For session coupling as as well as backend object
persistence.

Chris
 
G

gnari

Chris said:
I'm very surprised at this statement as well; mod_perl is "the bomb" --
it's rock solid.

I agree. mod_perl is pretty stable.
unless you do not use strict. then all bets are off.

gnari
 
T

thumb_42

Chris said:
We used perl, and regretted it for the web interface.
[snip]

If by "we used Perl" you mean you used something like HTML::Template or
Template Toolkit, then I am surprised at your statement. IMO,
HTML::Template coupled with Perl, separates the logic from the
presentation about as well as anything and provides much flexibilty and
power. I much prefer this technique to PHP. It sounds like you didn't
use a templating solution.

We used a template system, I'll admit, the template system wasn't ideal, but
that was not where the problems were. The template system did keep the
logic away from presentation, but it was awful to use because it was _very_
strict about how it did it, NO perl allowed in the templates at all,
everything was either static HTML or had to come from a function outside of
template space. I perfer the jsp syntax, (and because I like perl, later
wrote a template system using JSP syntax w/out the tags, but it wasn't
available for this project.)
I'm not trying to sound bigoted because this is c.l.p.m. I'm *in*
c.l.p.m because I've *been* bigoted *by* Perl. (I hope you see the
difference). I've used a lot of so called "web solutions" out there --
ASP, PHP, JSP, CF, Ruby, even .NET etc. -- and I keep coming back to
Perl. It rocks them all AFAIC.

I quite like perl too actually. Even though I'd been burnt by mod_perl (this
was 4+ years ago) I still might try and find some way to use it because I
really do like it.
I'm very surprised at this statement as well; mod_perl is "the bomb" --
it's rock solid. mod_perl on an Apache server can do ANYTHING. I'm not
trying to cast a shadow on you or the resources you had available to
you, but I can't help but express surprise at some of your statements,
because entirely the opposite has been true for me.

It was 4 years ago. (During the tail of the dot-com era) mod_perl *was* nice
for the simple things, but when I used it for stuff like storing database
connections (and yes, I used Apache::DBI) I had terrible memory leaks. Tried
different variations of them but couldn't weed them out. Removed DBI and the
leaks went away.

Now I *know* others didn't have this problem, in fact, most people probably
didn't. So, maybe it had something to do with the other modules I was using,
maybe it had to do with the order they were loaded in, could have been
a buggy database driver, I really don't know. (was using DB2 if that matters)

We also employed DBM as part of the session handeling, again.. problem area
periodic segfaults on the web server. The purely-perl stuff was absolutely
wonderful, though.
Give me Perl/mod_perl, Apache, MySQL, and HTML::Template and I'll put
together a multi-tiered, multi-server, load-balanced web solution that
would better anything (probably faster) put together by Java, PHP or
JSP. My opinion, but I'll bet many here would second it.

I know they would. (well, less the MySQL bit, mysql is not for "real"
database work)

I'll remind you at this point that I was _very_ happy with perl for the
"middle tier" stuff, just not mod_perl. You would not believe the amount of
flack I got for using perl on this project (I was the project mgr.) But, one
person (me) was able to write 90% of the code on this. In Java we would have
had to use several java developers. So, yes, I can definately appreciate
perl. :)
The most pain-free solution I have come up with for "sessions" in a
load-balancing situation is to take this data to the back-end or add
this as a separate tier as well. The extra time and resources taken for
lookups is worth the pain it removes esp. if your back-end solution is
tuned properly.

Yes, if the cost of having extra servers do sessions is justified, it'd be
the best idea. A lot of it boiled down to "how much are sessions worth?" In
our case, they weren't worth that much. Have to be careful not to make a
central point of failure though, simply using NFS or MySQL would have been
by far the easiest.. until that server goes down.. :)

One thing I'd really like to see (and maybe this is done already, haven't
looked) is mod_webapp or some such working with a perl back-end much the way
servlet/tomcat does. I hate the java language but do like some of it's
ideas.

If there were were something akin to "out of process" servlets written in
perl I'd seriously consider it. In fact, if I had 4 months of spare time,
I'd code one myself becase it'd be a fun project. :) (Hmm.. maybe perl
compiled to support multiplicity would be a requirement, with support for
each 'webapp' getting it's own instance of perl?)

Hmm.. and if this 'perlette' were taken down in an orderly fashion it could
have the option of mirroring it's sessions some place so that when clients
are directed to other servlets, the sessions are restored.

Anyhow, we were talking about large scale perl apps. I'd hold my ground on
keeping perl away from apache. Both perl and apache are complex software
packages and binding perl to apache would be a lot like embedding perl into
an OS's kernel - (kernel == good), (perl == good), (perl + kernel == very bad)

If PHP keeps on it's current track, I'd say the same for it. It needs to
stay fairly simple or it'll create problems with apache.

Jamie
 
T

thumb_42

gnari said:
I agree. mod_perl is pretty stable.
unless you do not use strict. then all bets are off.

I partially agree.

use strict;

It's super important in any project that involves more than 1 file. (well,
that's my general policy, if it takes more than 1 package or more than 15
minutes of devel time, it deserves strict, and really should have a main()
sub.) The "15 minute rule" is due to the fact that if you wait longer than
that, you'll probably have a mess adding strict later on. Best to do it
right away if you're unsure. :)

I big project w/out strict is like building a house with gasoline treated
lumber. :)

Jamie
 
S

Steven N. Hirsch

It was 4 years ago. (During the tail of the dot-com era) mod_perl *was* nice
for the simple things, but when I used it for stuff like storing database
connections (and yes, I used Apache::DBI) I had terrible memory leaks. Tried
different variations of them but couldn't weed them out. Removed DBI and the
leaks went away.

Now I *know* others didn't have this problem, in fact, most people probably
didn't. So, maybe it had something to do with the other modules I was using,
maybe it had to do with the order they were loaded in, could have been
a buggy database driver, I really don't know. (was using DB2 if that matters)

There have been other comments on the net alluding to memory leaks with
certain versions of DBD::DB2. Did you report it to DB2 support? IIRC,
the DBI driver is a supported application.
 
T

thumb_42

Steven N. Hirsch said:
There have been other comments on the net alluding to memory leaks with
certain versions of DBD::DB2. Did you report it to DB2 support? IIRC,
the DBI driver is a supported application.

Yea, I noticed. :)

Actually, I was kind of thankful for the bug, because stripping out DBI from
the web front turned out to be one of the best moves. Whether you use
mod_perl OR php for web interface, I'd recommend avoiding any database
connection from the web interface. (Only reason I tried it was that it
seemed convenient for simple reports and the like)

When I left, it looked something that looks like this:
(standard n-tier I guess)

Apache[] <------> Backend[] <------> Database
| |
+-------------------+--> Directory[] services, SMTP[], etc..

The []'s mean "more than 1 of", with DNS giving revolving IP's to Apache.

(with Database[] if you can manage it, we couldn't justify it, $$$)

Isolating any database access to the backends means we could replace them
with backends that only support queries and queued inserts if for some
reason we had to take down the database. (To swap to another vendor for
example)

Storing core business logic in a backend is easier to change because the
backends are completely transparent to the users. With dynamic, directory
based configuration we were able to easily take them in and out of the loop.
(Directory could have been an rsync'd conf file, in our case it just
happened to be LDAP, we needed authentication anyway.)

The backends spoke to the front ends using custom XML, (the "schema" was
specifically designed to parse fast, avoiding cdata and using attributes
make parsing XML trivial and FAST with XML::parser, using xml meant we could
easily add functionality to a backend w/out disrupting existing
functionality and we could 'boxcar' several business-logic functions
together)

Obviously these services reside on physically different machines.

Had I left the DBI code in the web interface, this would not have been
practical, maintenance would have been harder. In a way I was glad for
that bug. :)

Jamie
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
474,432
Messages
2,571,681
Members
48,796
Latest member
Greg L.

Latest Threads

Top