Is Python suitable for a huge, enterprise size app?

I

Ivan Van Laningham

Hi All--

George said:
:

An idea that perhaps takes the best of both worlds is use java for the
high level architecture and static type interfaces, and write the bulk
of the implementation in jython. PSF has awarded a grant to make jython
catch up with cpython, and that's good news for making the transition
from java to python smoother to a large audience. Others may have more
to say on the pros and cons of going with java/jython instead of
cpython, but it seems a good compromise to me.

And another option that you'll find has a great deal of viability is
using Python in the build process. Python's _wonderful_ to glue things
together, and its i18n capabilities are outstanding; I handled all the
internationalization and mock translation processes for a project near
the size you're describing in a dozen languages without working up a
sweat. Python is easily called by Ant, it deals really well with
unicode & utf-8, can easily parse Java resource files and XML files, and
you can stick all this stuff together in a few hours to handle _really
big builds_.

It's been nearly a year since the project failed and the company riffed
60 people from our site besides me, and the java monkeys that are left
are still using my Python scripts, in the same state I left 'em. They
can run them but they can't modify them. And every time they have to
produce a new mock-translation build, they complain to their boss: "We
used to have someone here who LIKED to do this stuff! And who could fix
it when it broke! We laid him off WHY????"

;-)

Metta,
Ivan
----------------------------------------------
Ivan Van Laningham
God N Locomotive Works
http://www.andi-holmes.com/
http://www.foretec.com/python/workshops/1998-11/proceedings.html
Army Signal Corps: Cu Chi, Class of '70
Author: Teach Yourself Python in 24 Hours
 
K

Kay Schluehr

George said:
Personally, although I find nothing comes close to the clarity and
flexibility that python offers, in this case I would go with java, if
for nothing else, to be on the safe side.

"No manager got ever fired for buying IBM"
Think about it: if the
project fails (and that's quite likely for huge projects, not dark
humour) and you've done it in python, everyone will blame python and
you who chose it; java OTOH belongs in the realm of "standard business
practices", so in a way "it's ok" to screw up, you did what everybody
else does. Yet another failed java enterprise project, nothing to see
here.

Hi George, You probably know the tale about Achilles and the turtle?
Where do You think Achilles would be located in the race against the
turtle if Zeno would have convinced him that motion is impossible? I
always thought that we Germans were hopeless pessimists ;)

It's a very funny logical loop inside this kind of arguments: don't use
great Python for grand multimillion $ projects because if You fail
everyone trashes cute Python and claims that it is not reliable for
grand projects and therefore will not be used for those.

Ciao,
Kay
 
N

Neil Hodgson

Steve M:
... discusses the problem with memory allocation in CPython.
> Apparently CPython is not good at, or incapable of, releasing
> memory back to the operating system.

I'm fairly sure Java does not give memory back to the operating
system either. A quick search did not give a definitive answer but there
were complaints from people about not being able to do this.

Neil
 
C

Cameron Laird

.
.
.
Just because the app itself is very OO, it doesn't always follow that the
database level needs to be - there are lots and lots of problems for which
normal RDBMS tables & joins are a pretty darn good fit. (I'm not trying to
.
.
.
While all of Dave's follow-up seems to me accurate and pertinent,
this sentence in particular inspires me to write, "Me, too."
 
G

George Sakkis

Kay Schluehr said:
Hi George, You probably know the tale about Achilles and the turtle?
Where do You think Achilles would be located in the race against the
turtle if Zeno would have convinced him that motion is impossible? I
always thought that we Germans were hopeless pessimists ;)

Yeah, I know the tale; the difference is that the convincing arguments
there were a fallacy, while Pointy-Haired-Bosses are for real ;-) So
instead of calling this pessimism, I would put it somewhere between
cynicism and realism :)
It's a very funny logical loop inside this kind of arguments: don't use
great Python for grand multimillion $ projects because if You fail
everyone trashes cute Python and claims that it is not reliable for
grand projects and therefore will not be used for those.

I see what you're saying; however there's an 'else' branch in the
argument. That is, *if* such a grand project succeeds (possibly at a
much lower budget than predicted), it will similarly be a big boost for
marketing python. It's a high-risk / high-payoff bet. The reason I
would be cynic or pessimist (call it what you will) in this case is
mainly the inexperience implied in the OP's post ("I have been looking
and reading a lot about Python recently"). Now, if a decent percentage
of the multimillion $ bill would be invested in hiring 10 PSF members
with Alex Martelli for project lead, I would certainly reconsider my
statement about going with java <wink>.

George
 
P

Paul Rubin

Kay Schluehr said:
To answer Your initial question: there is probably no technical reason
against Python as a language or the CPython runtime. Both are very
stable and mature.

I'd like to agree with this but I just can't. Python is a great
language for prototyping and trying stuff out. It's fast and pleasant
to write. But you're right when you say that constant testing and
integration is absolutely necessary to develop anything substantial in
Python. That is not a virtue, it's a fact that you need all that
testing and correction because it's so easy to make errors and break
things. The implementation is not so solid either. I find I file a
few bug reports in sourceforge (or these days often don't even bother)
just about every time I code something new in Python. I don't know if
language bugs discovered per kLOC written is a normal metric, but for
CPython, in my experience, I'd have to say that number is rather high
as compared with other languages. Examples I've found in just the
past couple days:

1) the socket module messes with instance variables on socket
objects in a way that stops the obvious
class logging_socket(socket):
# version of socket that logs all output to the console
def send(self, mesg):
print 'socket sending', repr(msg)
socket.send(self, mesg)
from working properly. This cost me about an hour of head scratching
to figure out what was going on. Some might say nothing is wrong,
others might say the socket module could use some reorganization,
but I'd say it's a questionable feature of the Python language
that this kind of bug can exist at all. I'm not a Java expert but
I believe this particular bug would be impossible in Java.

2) If you want to display a gif image in tkinter, you can make a
frame object, put a canvas in it, and put a PhotoImage object
onto the canvas, which works fine. But if you use your own
subclass of frame (maybe to add your own gui stuff), for some
reason the gif doesn't display in the canvas. Also, if you run
your frame's mainloop in its own thread, the gif sometimes
doesn't display. I'm still trying to figure this one out. And
every incident like this means I'm spending my time debugging
the Python library instead of writing my own code.

The documentation is spotty (Tkinter? Hah.) and the runtime library's
idea of supporting some protocol or standard is often to implement
some simple cases and not bother with the more difficult details. So
users tend to look in the manual and say "yeah, Python supports
[whatever]" and then get bitten later when they find that the support
for [whatever] is quite incomplete (look at the socket module's SSL
support, which doesn't bother validating the certificates, for an
extreme example, but it's similar with urllib, imaplib, etc.) Python
advocates also tend to say "Python supports such-and-such" if there's
a 3rd party module that supports it (e.g. the standard distro has no
support whatsoever for SQL server connectivity, so the OP's megabuck
app's developers will have to deal with a different set of maintainers
for some ODBC class than for the stdlib). An old version of the
Twisted Matrix docs explained why Twisted reimplemented so much stuff
that was already in CPython, saying something like "it may look like
we're reinventing the wheel, but we found the old wheels were often
square and made of glue". Or as Jamie Zawinski said about X-Windows,
"using these toolkits is like trying to make a bookshelf out of mashed
potatoes."

Java's runtime library has a rather ugly and tasteless design compared
with Python's, but it's much more thorough (includes JDBC, JSSE, a
more serious threading implementation, etc.), the documentation
coverage is pretty good, and when they implement a standard, they do
tend to implement it completely.

I'm in the middle of prototyping something in Python and it's going
pretty well despite occasional snags like those mentioned above. But
at the end of the day I don't think what we get will be solid enough
for use in production. We're planning to use the Python prototype to
get the feature set and protocols figured out, then reimplement in Java.
The OP might consider that approach.
 
V

vincent wehren

| The company I work for is about to embark on developing a commercial
| application that will cost us tens-of-millions to develop. When all is
| said and done it will have thousands of business objects/classes, some
| of which will have hundreds-of-thousands of instances stored in a DB.
| Our clients will probably have somewhere between 50-200 users working
| on the app during the day, possibly in mutiple offices, and then a
| large number of batch processes will have to run each night. It will
| also need to have a web interface for certain users. It needs to be
| robust, easy to maintain, and able to be customized for each client.
|
| Right now it looks like Java is the language of choice that the app
| will be developed in. However, I have been looking and reading a lot
| about Python recently and it seems to me that Python could handle it.

If the application is really huge, you should consider using
a set of programming languages that do the particular job best.
That is, go for a complementary, concurrent approach in your choice
of programming languages. I wouldn't want to religiously stick to any
one language just for the sake of being religious.

I am sure that from a "best-for-the-job" perspective, Python is a serious
candidate for most of them. The aspects robust, easy to maintain, and
suitable for customization that you envision as properties
for your application are properties that need to be matched the
language(s) of choice, as well. Again, here Python is a more than serious
contender.


Regards,
--

Vincent Wehren





| The big attraction to me is the developer productivity. It seems that
| if Python can handle it, then we could gain a huge savings by using
| Python instead of Java from a productivity standpoint alone.
|
| So, given the very general requirements in the first paragraph, do you
| think that Python could handle it? If anyone has direct experience
| developing large apps in Python, I would appreciate your insight.
| Based on the responses I get, I am planning on writing a proposal to my
| management to consider Python instead of Java.
|
| Thanks,
| John
|
 
F

Fredrik Lundh

Steve M said:
http://mail.python.org/pipermail/python-dev/2005-January/051255.html

discusses the problem with memory allocation in CPython. Apparently
CPython is not good at, or incapable of, releasing memory back to the
operating system.

and unless your operating system is totally braindead, and thus completely unfit
to run huge enterprise size applications, that doesn't really matter much. leaks
are problematic, large peak memory use isn't.

(fwiw, I'm tempted to argue that people who don't understand how memory
allocation works on modern systems are unfit to run enterprise applications as
well, but that's another story).

btw, very few memory allocators "release memory back to the operating system"
for small allocations; most memory allocators, including Pythons, do that for large
allocations. try running this, and check the memory use after each step:
x = 10000000 * "x"
y = x[1:]
z = x[2:]
del z
del y
del z

still claiming that CPython's incapable of releasing memory back to the system?

(finding the smallest multiplier that gives the same effect is left as an exercise etc.)

</F>
 
B

bruno modulix

Ivan Van Laningham wrote:
(snip)
What you're going to run into are two major stumbling blocks. One,
Python's got no credibility with management types unless the
credibility's already there. "Python? Never heard of it. Tell me
about it. ... Oh, it's interpreted, is it? Interesting."

Nope, Sir, it's byte-compiled just like Java. But you don't have to
invoke the compiler yourself...
You can
see Python going down the sewer pipes, right on their faces. Two,
security. "This python sounds pretty interesting. Tell me about the
security. How can we prevent people from stealing our source code,
which we just spent millions developing? ... Hmm, trust the developers
out there not to peek? Oh, sure, let's use it."

Just like Java, which is so easy to reverse-engineer...
(True, there are ways
around the second, but you're going to have to talk _very_ fast and have
ALL the answers before the management type gets to his/her office and
shuts the door in your face and on your idea.)

+1
 
F

Fredrik Lundh

Paul said:
2) If you want to display a gif image in tkinter, you can make a
frame object, put a canvas in it, and put a PhotoImage object
onto the canvas, which works fine. But if you use your own
subclass of frame (maybe to add your own gui stuff), for some
reason the gif doesn't display in the canvas.

did you read the PhotoImage documentation? see e.g. the note at the
bottom of this page:

http://effbot.org/zone/tkinter-photoimage.htm

(this used to be explained in the Python FAQ too, but it looks as if that
was lost in the great FAQ reorganization).

</F>
 
P

Paul Rubin

Fredrik Lundh said:
did you read the PhotoImage documentation? see e.g. the note at the
bottom of this page:

http://effbot.org/zone/tkinter-photoimage.htm

Thanks! I hadn't seen that. It explains what otherwise seemed
completely bizarre.
(this used to be explained in the Python FAQ too, but it looks as if that
was lost in the great FAQ reorganization).

I wouldn't have thought of looking there either. Do you think you
could mention it in your Tkinter doc? That would be a logical place
for it.
 
P

Paul Rubin

Paul Rubin said:
... Do you think you
could mention it in your Tkinter doc? That would be a logical place
for it.

Ehhh, specifically I meant it would be good to mention this in the
PhotoImage example in Chapter 16 of "An Introduction to Tkinter" (The
Canvas Image Item). That's the example I followed and was confused as
heck why the image only appeared some of the time (I never saw it
actually appear and then disappear, but that's just luck).

Thanks again, I never would have figured this one out.
 
M

Magnus Lycka

john67 said:
The company I work for is about to embark on developing a commercial
application that will cost us tens-of-millions to develop. When all is
said and done it will have thousands of business objects/classes,
Ok, that sounds like a big app, but does it really have to be like
that, or is that something you imagine because things usually get
that complex using other tools?

In my experience, development costs grow faster than linearly with
class model size. (Of course, this is true regardless of programming
language.)

Prototyping things with Python is often an good way to discover simpler
ways of handling a problem, and it's not unheard of that a Python
prototype is good enough to keep as the final implementation.

Thus, apart from the fact that you typically meed much less code to
develop the same functionality in Python than in Java, you might also
discover that you can do things in a smarter way which reduces you
application size even more.

There are plenty of descriptions of successful use of Python at:
http://www.pythonology.com/

Of course, every language has a sweetspot, and particularly if you
have a bunch of experienced Java developers who will think in Java
idioms whatever language they code in, they might end up building
a really stupid system in Python. Perhaps you should also read:
http://dirtsimple.org/2004/12/python-is-not-java.html
http://dirtsimple.org/2004/12/java-is-not-python-either.html
some
of which will have hundreds-of-thousands of instances stored in a DB.
That sounds like a moderately sized database.Hundreds of thousands of
new entries per month might be big if you need to keep data for years,
but this doesn't seem to be a limiting factor for you assuming a system
which uses a real database backend.
Our clients will probably have somewhere between 50-200 users working
on the app during the day, possibly in mutiple offices, and then a
large number of batch processes will have to run each night.

This is not a problem in itself. There are a number of different ways
to partition such a system between clients and different kinds of
servers, but it can certainly be done in Python.
It will
also need to have a web interface for certain users. It needs to be
robust, easy to maintain, and able to be customized for each client.

This sounds like a good fit. Python is at least excellent for the
customization part. I'll talk about how we do that here at Carmen
Systems at the Europython conference in June. Slides will be available.
Mail me if you want to have a look at a draft.

We basically use Python for customization and use C++ for the core code.
Python and C++ is a very good mix. Java and CPython might not be such
a good fit, but that depends a bit what your intentions with Java is.
(E.g. is JNI ok, or is hardware independence considered very important?)

Java/Jython is a better pair of course, but Jython has few maintainers
and is a few versions behind CPython.
Right now it looks like Java is the language of choice that the app
will be developed in. However, I have been looking and reading a lot
about Python recently and it seems to me that Python could handle it.

To be perfectly honest, I'd be very wary to stick out my neck and be
a Python champion if my trust in Python was based only on reading...
As a manager I'd also find it rather unconvincing...

Let me put it this way: Python has a potential to make your system
better, and to reach your goals in a shorter time. It fits very well
with an agile approach, and in most systems of the kind I imagine you
intend to build, agile development is very helpful in tracking project
progress. You produce *real* programs early, not a lot of paperwork
which a year later turns out to represent a misunderstanding of the
actual problem, or how an effective solution should behave.

I think it would be worthwile to investigate what Python could mean to
you. It might certainly be very useful in the toolbox of the developers.
It might be just a tool to aid in the development, it might be used as
a glue language to connect interacting components written in other
languages (Java, C++, C# etc), it might be an embedded macro language,
a prototyping tool, or the implementation language for at least parts
of the application.

After all, one tool is unlikely to fit all tasks.

Why not hire a well respected Python trainer for a few days, let a few
developers learn Python and explore how it could fit into your system.
The big attraction to me is the developer productivity. It seems that
if Python can handle it, then we could gain a huge savings by using
Python instead of Java from a productivity standpoint alone.

Quite possible, but the choice of programming language(s) is just one
of many very important decisions that determine the outcome of a
software development project.
So, given the very general requirements in the first paragraph, do you
think that Python could handle it? If anyone has direct experience
developing large apps in Python, I would appreciate your insight.

You might want to consult with e.g. Marc-Andre Lemburg, who has
experience in developing large business apps in Python. (See
http://www.egenix.com/ )
Based on the responses I get, I am planning on writing a proposal to my
management to consider Python instead of Java.

Consider suggesting it as a complement.
 
V

Ville Vainio

brian> to build expertise and confidence, etc. People are more
brian> important than programming language.

Or so you would think before writing sizeable software in C++.
 
M

Maurice LING

Just like Java, which is so easy to reverse-engineer...
It makes big difference (legally) to if the codes are there and someone
sees it, to if the codes are locked in some packaged or zipped form and
someone reverse-engineer it. It is legally as different as if you drop
money on the ground and I pick it up, to pick-pocketing you and take the
money.

Nobody seems to be able to understand this simple logic.

Yes, Java class files can be reverse-engineered quite easily but the act
of doing that is criminal, unless performed under specified Council
Directives (in EU) or under any statutory law in specific countries. But
the act of looking at the codes if it is there is not criminal, just
like reading a book in bookstore. If anyone can program in binary today,
no codes will be safe with them anyway...

maurice
 
R

Roy Smith

George Sakkis said:
Another issue that hasn't been mentioned so far is staffing. You can
find java programmers and software engineers a dime dozen, but for
python it will be harder.

Yes, there are a lot more people out there who put Java on their resume
than put Python on their resume. Most of those so-called Java programmers
write total crap. Of course, there's a lot of crap written in Python too.
But since there're a lot more Java programmers out there, it stands to
reason that there's a lot more crappy Java programmers too.

It's a lot easier to teach a good programmer Python than it is to teach a
crappy programmer good software engineering skills.
 
C

Cameron Laird

.
.
.
If the application is really huge, you should consider using
a set of programming languages that do the particular job best.
That is, go for a complementary, concurrent approach in your choice
of programming languages. I wouldn't want to religiously stick to any
one language just for the sake of being religious.
.
.
.
How embarrassing--I've followed up twice in this thread
already, and utterly failed to mention this. YES! The
question should never be, "What one language is least
imperfect for our requirements?", but rather, "What
specific combination of languages best fits our
situation?" I'll refer again to Phaseit's deliveries:
we *always* are working in multiple languages. If we
had responsibility for the project the Original Poster
described, I suspect we'd involve Python and C++, or
Jython and Java, and of course SQL and XHTML, and ...

At this level, part of my endorsement of Python has to
do with its propensity to play nicely with others.

As others have written, though, it would be rather
.... ambitious to commit tens of millions of dollars to
a language with which the organization is almost
entirely unfamiliar. 'Course, Big Organizations do
that all the time, without realizing it ...
 
P

Peter Hansen

Maurice said:
It makes big difference (legally) to if the codes are there and someone
sees it, to if the codes are locked in some packaged or zipped form and
someone reverse-engineer it. It is legally as different as if you drop
money on the ground and I pick it up, to pick-pocketing you and take the
money.

Nobody seems to be able to understand this simple logic.

So you're saying that reverse engineering Java bytecode is illegal,
while doing the same with Python bytecode is not? Or something like
that? (And you're a lawyer, right? Because if you're not, and you're
not citing your sources, why is it we should put any value in these
comments about what is (legally) true?)

-Peter
 
D

Dave Brueck

Paul said:
Kay Schluehr said:
To answer Your initial question: there is probably no technical reason
against Python as a language or the CPython runtime. Both are very
stable and mature.


I'd like to agree with this but I just can't. Python is a great
language for prototyping and trying stuff out. It's fast and pleasant
to write. But you're right when you say that constant testing and
integration is absolutely necessary to develop anything substantial in
Python. [snip]
I'm in the middle of prototyping something in Python and it's going
pretty well despite occasional snags like those mentioned above. But
at the end of the day I don't think what we get will be solid enough
for use in production. We're planning to use the Python prototype to
get the feature set and protocols figured out, then reimplement in Java.
The OP might consider that approach.

You bring up some good points and I imagine the bugs you've encountered are
pretty frustrating, but I'm surprised at how different our experience has been.
I know my comments about Python probably come across a "total fan boy" - but for
us Python has been extremely positive.

We currently use Python, in production, in the following ways:

- custom HTTP web servers
- custom HTTP proxies
- behind-the-scenes log processors and other operational tools
- several Zope-based applications, both internally and externally facing
- both the client and server sides of a distributed work cluster
- the client side of a media encoding farm (server is a Zope app)
- a downloadable Windows client that our customer's customers use, that includes
a Windows COM server (an ActiveX control we implemented using ctypes).

Most of these have been in production for a year or more, with the oldest being
about 4 years (not a long time, but certainly long enough to be considered
stable). The toughest pieces initially were the Zope apps and the Windows COM
stuff - Zope because of documentation and COM because, well, it's COM.

During that time, the biggest problem we had with Python and its standard
libraries was one issue with socket.recv and memory allocation when doing lots
of receives. That lost us a few days of hunting for a memory leak, but we were
able to resolve it by looking at the C source. Another problem was the need to
call unlink (or something like that) on XML minidom elements in some
circumstances. Apart from that, we haven't encountered much of anything in terms
of bugs in the language or the standard library in *years*. In fact, I don't
recall encountering a bug in the language itself.

We've generally trailed the standard Python releases by about a year (although
this week I moved the encoding farm to Python 2.4.1, which is half a year
early). We don't use Tkinter. We don't use much outside the standard library: we
make heavy use of ctypes, psycopg (Postgres), and lately CherryPy, but that's
about it. So for us, the number of bugs in Python + stdlib has been really low.

One thing from your experience that did resonate with me is that, except for
ftplib and occasionally urllib (for basic, one-shot GETs), we don't use any of
the standard library's "protocol" modules - partly because we had to implement
our own HTTP libraries for performance and scalability reasons anyway, and
partly because we had trouble figuring out e.g. all the ins and outs of
urllib/urllib2/httplib.

Overall it's been such a positive experience for us that nobody in the company -
from grunt testers up to the CTO - has any reservations about using Python in
production anymore (even though initially they all did). All of the developers
have previous experience with using Java in production systems, and none
seriously consider it for new projects at all.

-Dave
 
A

Aahz

(this used to be explained in the Python FAQ too, but it looks as if that
was lost in the great FAQ reorganization).

Send e-mail to (e-mail address removed) -- actually, you can have commit
privs if you want.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,481
Members
44,900
Latest member
Nell636132

Latest Threads

Top