Is Python suitable for a huge, enterprise size app?

[john67]

The company I work for is about to embark on developing a commercial
application that will cost us tens-of-millions to develop. When all is
said and done it will have thousands of business objects/classes, some
of which will have hundreds-of-thousands of instances stored in a DB.
Our clients will probably have somewhere between 50-200 users working
on the app during the day, possibly in mutiple offices, and then a
large number of batch processes will have to run each night. It will
also need to have a web interface for certain users. It needs to be
robust, easy to maintain, and able to be customized for each client.

Right now it looks like Java is the language of choice that the app
will be developed in. However, I have been looking and reading a lot
about Python recently and it seems to me that Python could handle it.
The big attraction to me is the developer productivity. It seems that
if Python can handle it, then we could gain a huge savings by using
Python instead of Java from a productivity standpoint alone.

So, given the very general requirements in the first paragraph, do you
think that Python could handle it? If anyone has direct experience
developing large apps in Python, I would appreciate your insight.
Based on the responses I get, I am planning on writing a proposal to my
management to consider Python instead of Java.

Thanks,
John

 

[bplumhoff]

The answer seems to be yes. See http://www.artima.com/intv/speed.html

HTH,
Bernd

 

[keirr]

So, given the very general requirements in the first paragraph, do
you

think that Python could handle it? If anyone has direct experience
developing large apps in Python, I would appreciate your insight.

I wouldn't, especially[1] if your thousands of business objects get
allocated/deallocated as the system runs. Currently python's memory
usage can grow rapidly (from the perspective of the o/s) when large
numbers of objects are repeatedly created and freed.

On the other hand, you could use a combination of python and C++
(perhaps using boost's python wrapping code). The ability to create
C/C++ modules is a "get out of jail free card" - if you did manage to
convince your company to go with python, and it fell short in some
area, a custom C module could be a handy trick.

Perhaps you could compare your project to existing large python
projects; like Zope maybe?

[1] if the objects in the system are static, rather than dynamic, of
course this is not an issue. My "I wouldn't" comes from the fact that
your requirements include "lots of object" as a feature, and no word on
how they will be used.

Good luck anyway
Keir.

 

[Larry Bates]

I don't see anything listed that would eliminate Python. You
may want to take a quick look at some of the rather large
applications that have been done with Zope (written in Python).
Zope might even be a good application platform to build upon
(can't say for sure, not enough detail about application).
Even if you don't want to use all of Zope the ZODB is a well
tested persistent Python object store that might prove
helpful.

There are many other projects supporting WAY more than 200
users that are written in Python.

-Larry Bates

 

[john67]

Would the allocation/deallocation memory usage issue be different than
it would be with Java? Both Python and Java have automatic garbage
collection, correct? Is Python's not as effective as Java's? I think
the memory issues for the two languages would be similar in this area,
but maybe I am wrong.

John

 

[Peter Dembinski]

[snap]

I wouldn't, especially[1] if your thousands of business objects get
allocated/deallocated as the system runs. Currently python's memory
usage can grow rapidly (from the perspective of the o/s) when large
numbers of objects are repeatedly created and freed.

Isn't it true that in recent Python releases, one may replace the
default GC with custom one?

On the other hand, you could use a combination of python and C++
(perhaps using boost's python wrapping code). The ability to create
C/C++ modules is a "get out of jail free card" - if you did manage
to convince your company to go with python, and it fell short in
some area, a custom C module could be a handy trick.

In fact, it is not a trick -- combining C with Python is fairly
approved way of speeding up the app

 

[keirr]

I wouldn't, especially[1] if your thousands of business objects get

Isn't it true that in recent Python releases, one may replace the
default GC with custom one?

Good point. Indeed, since you have the source code you can change
anything you like. My warning was aimed at a specific situation, and
was meant to be a 'keep this in mind' comment (I phrased it as "don't
use python" really because I expected the majority of the responses to
be "do use python", and thought a bit of balance would be helpful).

As for a custom garbage collector, there has been a little discussion
about the gc code (on this group). My feeling is, if it was easy to
make it better someone would have done it already. Note I don't say
it's impossible (particularly with the resources the op mentioned),
just not straight-forward.

Of course, I wouldn't use Java, but that's another story '-)

Cheers,

Keir.

 

[Dave Brueck]

The company I work for is about to embark on developing a commercial
application that will cost us tens-of-millions to develop. [snip]
Right now it looks like Java is the language of choice that the app
will be developed in. However, I have been looking and reading a lot
about Python recently and it seems to me that Python could handle it.
The big attraction to me is the developer productivity. It seems that
if Python can handle it, then we could gain a huge savings by using
Python instead of Java from a productivity standpoint alone.

So, given the very general requirements in the first paragraph, do you
think that Python could handle it?

Yes.

Python has good database support, it works well on a wide range of platforms,
and it's great at tying together different processes, machines, etc. - for
example, it's fairly easy to get Python to access C code, dynamic libraries,
system APIs, and external programs. It's easier to test than Java code as well,
and the overal cost of change is lower.

The concise nature of the language means that what would otherwise be a "huge"
enterprise app in Java might just be a "large" enterprise app in Python - the
project will not reach that unwieldy size as quickly, if ever.

The Google archives of this group have this topic covered in various ways many
times over - IMO Python is great for smallish apps, but its advantage over e.g.
Java actually *increases* with the size of the app.

If you go the Python route, two of the main obstacles will be:

1) mindshare - getting people on board, getting them to overcome biases one way
or another, convincing them to really take the time to come up to speed on Python.

2) reducing the complexity of what you think you need to build. I don't know how
to better describe this, but in Java, for example, I'd have whole bunches custom
classes to do various tasks but the Python equivalent collapsed into a single
module. Anyway, I predict that it'll take you some time to convince yourself
that you simply won't need to build all of the same components as you otherwise
would, or that they'll often be vastly simpler.

-Dave

 

[Cameron Laird]

The company I work for is about to embark on developing a commercial
application that will cost us tens-of-millions to develop. When all is
said and done it will have thousands of business objects/classes, some
of which will have hundreds-of-thousands of instances stored in a DB.
Our clients will probably have somewhere between 50-200 users working
on the app during the day, possibly in mutiple offices, and then a
large number of batch processes will have to run each night. It will
also need to have a web interface for certain users. It needs to be
robust, easy to maintain, and able to be customized for each client.

Right now it looks like Java is the language of choice that the app
will be developed in. However, I have been looking and reading a lot
about Python recently and it seems to me that Python could handle it.
The big attraction to me is the developer productivity. It seems that
if Python can handle it, then we could gain a huge savings by using
Python instead of Java from a productivity standpoint alone.

So, given the very general requirements in the first paragraph, do you
think that Python could handle it? If anyone has direct experience
developing large apps in Python, I would appreciate your insight.
Based on the responses I get, I am planning on writing a proposal to my
management to consider Python instead of Java.

.
.
.
My flippant response is that you'll need to choose Java if
you want it to cost that much.

Phaseit regularly develops applications that serve *thousands*
of users, with database record counts in the millions, and our
total costs are typically in the tens of thousands of dollars,
rather than tens of millions.

On the other hand, maybe I don't want you doing this in Python.
Most enterprise projects fail--or at least there's plausible
evidence to believe that most fail--and perhaps I shouldn't
encourage Python's use in something that will fail.

 

[Steve M]

This thread:

http://mail.python.org/pipermail/python-dev/2005-January/051255.html

discusses the problem with memory allocation in CPython. Apparently
CPython is not good at, or incapable of, releasing memory back to the
operating system. There are ways to compensate for this. I guess the
comment about C modules was meant as one way to do so, either by
reducing memory requirement in the first place (C data structures are
more compact than Python) or else by allocating and freeing memory
wholly in the C module, which perhaps does work as expected.

The web page for Evan Jones does not obviously indicate the status of
the improved memory allocator patch he was working on. I wonder if it
is coming along.

Incidentally, does anyone know the prospects for CPython to be made
stackless? Say in 2.5 or 2.9? Or will that always be an independent
project?

 

[john67]

LOL! That is really funny, in a dark humor kind of way. I don't want
the project to fail either. I am not convinced that we will succeed if
we go the Java route. However, I am just a grunt in the chain and not
in a position to make the decision. I hope I can have some influence
on the decision though. I will be pleased if I can get someone to at
least seriously consider Python.

I understand your sentiment about not encouraging something that might
fail to be done in Python. Hopefully it won't come to that. Thanks
for the info on Phaseit apps.

John

 

[EHP]

Python has good database support, it works well on a wide range of

platforms, and it's great at tying together different processes, machines,
etc. - for example, it's fairly easy to get Python to access C code,
dynamic libraries, system APIs, and external programs. It's easier to test
than Java code as well, and the overal cost of change is lower.

Python has good DB support - but only simple connectors (like JDBC).
ORM in Java (like Hibernate) are much better than ORM in Python (probably the
best is sqlobject). How you can write huge OO app without ORM ?
IMHO supporting libraries for bussiness apps are better in Java (don't flame
please).

EHP

 

[Ivan Van Laningham]

Hi All--

LOL! That is really funny, in a dark humor kind of way. I don't want
the project to fail either. I am not convinced that we will succeed if
we go the Java route. However, I am just a grunt in the chain and not
in a position to make the decision. I hope I can have some influence
on the decision though. I will be pleased if I can get someone to at
least seriously consider Python.

I understand your sentiment about not encouraging something that might
fail to be done in Python. Hopefully it won't come to that. Thanks
for the info on Phaseit apps.

What you're going to run into are two major stumbling blocks. One,
Python's got no credibility with management types unless the
credibility's already there. "Python? Never heard of it. Tell me
about it. ... Oh, it's interpreted, is it? Interesting." You can
see Python going down the sewer pipes, right on their faces. Two,
security. "This python sounds pretty interesting. Tell me about the
security. How can we prevent people from stealing our source code,
which we just spent millions developing? ... Hmm, trust the developers
out there not to peek? Oh, sure, let's use it." (True, there are ways
around the second, but you're going to have to talk _very_ fast and have
ALL the answers before the management type gets to his/her office and
shuts the door in your face and on your idea.)

Metta,
Ivan
----------------------------------------------
Ivan Van Laningham
God N Locomotive Works
http://www.andi-holmes.com/
http://www.foretec.com/python/workshops/1998-11/proceedings.html
Army Signal Corps: Cu Chi, Class of '70
Author: Teach Yourself Python in 24 Hours

 

[Kay Schluehr]

Would the allocation/deallocation memory usage issue be different than
it would be with Java? Both Python and Java have automatic garbage
collection, correct?

In recent Python versions the CPython interpreter offers a
cycle-collector which weakens the most profound counter argument
against reference-counting techniques.

To answer Your initial question: there is probably no technical reason
against Python as a language or the CPython runtime. Both are very
stable and mature. Extension and optimization techniques are well
understood. The library support is great. I would be more concerned
about the development strategy. Be aware that Python is weak in
defining constraints on interfaces even more than Java ( Javas static
type system makes a bit easier what doesn't mean that static typing is
a really adequate solution for it at all ). There are almost no
declarative elements in the language. You have somehow to think about
communicating contracts between different developers across the team.
Elements of lightweight methodologys like continous integration and
early unit-testing are mandatory, not optional. Be also aware that
there are no IDEs / GUI-builders and UML-designers which are comparable
to those for Java or dotNET. You won't come as close to a group
consensus as if You would stick to VisualStudio7 for C#, or
IDEA/Eclipse for Java.

Regards,
Kay

 

[Dave Brueck]

Python has good DB support - but only simple connectors (like JDBC).
ORM in Java (like Hibernate) are much better than ORM in Python (probably the
best is sqlobject). How you can write huge OO app without ORM ?

I don't doubt that Hibernate is more mature, but having not used it, I can't
really say how much better/worse it is than, say, sqlobject for Python. But, as
you pointed out, there *are* ORM libraries for Python.

FWIW, the only times I've been involved in database-centric projects with a cost
in the "tens of millions", the database team steered clear of any sort of any
automatic ORM layer altogether. Depending on your application, an ORM can fall
into the same category as, say, EJB - a nifty idea that looks great on paper but
can cause more problems than it solves.

Just because the app itself is very OO, it doesn't always follow that the
database level needs to be - there are lots and lots of problems for which
normal RDBMS tables & joins are a pretty darn good fit. (I'm not trying to
discount ORM layers or even object databases, just disagreeing with the notion
that huge OO apps automatically require ORM)

Have fun,
-Dave

 

[Jack Diederich]

The company I work for is about to embark on developing a commercial
application that will cost us tens-of-millions to develop. When all is
said and done it will have thousands of business objects/classes, some
of which will have hundreds-of-thousands of instances stored in a DB.
Our clients will probably have somewhere between 50-200 users working
on the app during the day, possibly in mutiple offices, and then a
large number of batch processes will have to run each night. It will
also need to have a web interface for certain users. It needs to be
robust, easy to maintain, and able to be customized for each client.

[snip]

The technical specifications seem OK, these aren't language constraints
in python more than any other language. The people constraints...

Tens of millions to develop? Whoa - it is easy for me to predict absolute
and complete failure right off the bat no matter the language (but only
after incurring cost overruns). If your head is anywhere near the chopping
block I suggest proffering a "proof of concept" version first. If you
can't do a first version in six months with a team of six people it is a sign
that you don't really know what you want. The final version might be
bigger and have more features - but 10M of features is pie in the sky stuff.
Everyone involved must have wild dreams about how awesome the final version
will be, and all those dreams diverge.

Ten million is ten guys writing for two years (top notch at US prices) to come
up with a product from sratch. Tens of millions is? Best of luck.

-jackdied

 

[Andrew Dalke]

... Oh, it's interpreted, is it? Interesting." You can
see Python going down the sewer pipes, right on their faces.

Nahh, the right answer is "It's byte-compiled, just like Java."

Andrew
(e-mail address removed)

 

[George Sakkis]

What you're going to run into are two major stumbling blocks. One,
Python's got no credibility with management types unless the
credibility's already there. "Python? Never heard of it. Tell me
about it. ... Oh, it's interpreted, is it? Interesting." You can
see Python going down the sewer pipes, right on their faces. Two,
security. "This python sounds pretty interesting. Tell me about the
security. How can we prevent people from stealing our source code,
which we just spent millions developing? ... Hmm, trust the developers
out there not to peek? Oh, sure, let's use it." (True, there are ways
around the second, but you're going to have to talk _very_ fast and have
ALL the answers before the management type gets to his/her office and
shuts the door in your face and on your idea.)

Another issue that hasn't been mentioned so far is staffing. You can
find java programmers and software engineers a dime dozen, but for
python it will be harder. The alternative of teaching python to a team
for a week or a month is IMO far too risky when you're talking about
millions; even if they are able to read or code fast in a new language,
they won't have grasped the idioms and the good programming techniques,
so they will be essentially mentally translating from java/c++/c#,
whatever their programming mother tongue is. On the bright side, in
python you won't have to try as hard to pick the few good programmers
among a sea of average/inexperienced/clueless ones that you'll have for
java.

Personally, although I find nothing comes close to the clarity and
flexibility that python offers, in this case I would go with java, if
for nothing else, to be on the safe side. Think about it: if the
project fails (and that's quite likely for huge projects, not dark
humour) and you've done it in python, everyone will blame python and
you who chose it; java OTOH belongs in the realm of "standard business
practices", so in a way "it's ok" to screw up, you did what everybody
else does. Yet another failed java enterprise project, nothing to see
here.

An idea that perhaps takes the best of both worlds is use java for the
high level architecture and static type interfaces, and write the bulk
of the implementation in jython. PSF has awarded a grant to make jython
catch up with cpython, and that's good news for making the transition
from java to python smoother to a large audience. Others may have more
to say on the pros and cons of going with java/jython instead of
cpython, but it seems a good compromise to me.

George

 

[brian]

[...] that will cost us tens-of-millions to develop.
[...]
Right now it looks like Java is the language of choice that the app
will be developed in. However, I have been looking and reading a lot
about Python recently [...]

Ignoring the technical aspects: if you have good group of people that
already know Java, and don't know Python, then use Java for this huge
project. Do a smaller project in Python first for the learning curve,
to build expertise and confidence, etc. People are more important than
programming language.

Brian.

 

[Dave Brueck]

Personally, although I find nothing comes close to the clarity and
flexibility that python offers, in this case I would go with java, if
for nothing else, to be on the safe side. Think about it: if the
project fails (and that's quite likely for huge projects, not dark
humour) and you've done it in python, everyone will blame python and
you who chose it; java OTOH belongs in the realm of "standard business
practices", so in a way "it's ok" to screw up, you did what everybody
else does. Yet another failed java enterprise project, nothing to see
here.

lol - I got a kick out of this post and I agree with it: *if* you're pretty sure
the project will ultimately fail, and you're planning for the best way to point
fingers and avoid the ax when it happens, then by all means, Java is the right
way to go. Java is the new "nobody ever got fired for buying IBM".

-Dave