Is Python suitable for a huge, enterprise size app?

Discussion in 'Python' started by john67, May 18, 2005.

  1. john67

    john67 Guest

    The company I work for is about to embark on developing a commercial
    application that will cost us tens-of-millions to develop. When all is
    said and done it will have thousands of business objects/classes, some
    of which will have hundreds-of-thousands of instances stored in a DB.
    Our clients will probably have somewhere between 50-200 users working
    on the app during the day, possibly in mutiple offices, and then a
    large number of batch processes will have to run each night. It will
    also need to have a web interface for certain users. It needs to be
    robust, easy to maintain, and able to be customized for each client.

    Right now it looks like Java is the language of choice that the app
    will be developed in. However, I have been looking and reading a lot
    about Python recently and it seems to me that Python could handle it.
    The big attraction to me is the developer productivity. It seems that
    if Python can handle it, then we could gain a huge savings by using
    Python instead of Java from a productivity standpoint alone.

    So, given the very general requirements in the first paragraph, do you
    think that Python could handle it? If anyone has direct experience
    developing large apps in Python, I would appreciate your insight.
    Based on the responses I get, I am planning on writing a proposal to my
    management to consider Python instead of Java.

    Thanks,
    John
     
    john67, May 18, 2005
    #1
    1. Advertising

  2. john67

    Guest

    , May 18, 2005
    #2
    1. Advertising

  3. john67

    keirr Guest

    > So, given the very general requirements in the first paragraph, do
    you
    > think that Python could handle it? If anyone has direct experience
    > developing large apps in Python, I would appreciate your insight.



    I wouldn't, especially[1] if your thousands of business objects get
    allocated/deallocated as the system runs. Currently python's memory
    usage can grow rapidly (from the perspective of the o/s) when large
    numbers of objects are repeatedly created and freed.

    On the other hand, you could use a combination of python and C++
    (perhaps using boost's python wrapping code). The ability to create
    C/C++ modules is a "get out of jail free card" - if you did manage to
    convince your company to go with python, and it fell short in some
    area, a custom C module could be a handy trick.

    Perhaps you could compare your project to existing large python
    projects; like Zope maybe?

    [1] if the objects in the system are static, rather than dynamic, of
    course this is not an issue. My "I wouldn't" comes from the fact that
    your requirements include "lots of object" as a feature, and no word on
    how they will be used.

    Good luck anyway :)
    Keir.
     
    keirr, May 18, 2005
    #3
  4. john67

    Larry Bates Guest

    I don't see anything listed that would eliminate Python. You
    may want to take a quick look at some of the rather large
    applications that have been done with Zope (written in Python).
    Zope might even be a good application platform to build upon
    (can't say for sure, not enough detail about application).
    Even if you don't want to use all of Zope the ZODB is a well
    tested persistent Python object store that might prove
    helpful.

    There are many other projects supporting WAY more than 200
    users that are written in Python.

    -Larry Bates

    john67 wrote:
    > The company I work for is about to embark on developing a commercial
    > application that will cost us tens-of-millions to develop. When all is
    > said and done it will have thousands of business objects/classes, some
    > of which will have hundreds-of-thousands of instances stored in a DB.
    > Our clients will probably have somewhere between 50-200 users working
    > on the app during the day, possibly in mutiple offices, and then a
    > large number of batch processes will have to run each night. It will
    > also need to have a web interface for certain users. It needs to be
    > robust, easy to maintain, and able to be customized for each client.
    >
    > Right now it looks like Java is the language of choice that the app
    > will be developed in. However, I have been looking and reading a lot
    > about Python recently and it seems to me that Python could handle it.
    > The big attraction to me is the developer productivity. It seems that
    > if Python can handle it, then we could gain a huge savings by using
    > Python instead of Java from a productivity standpoint alone.
    >
    > So, given the very general requirements in the first paragraph, do you
    > think that Python could handle it? If anyone has direct experience
    > developing large apps in Python, I would appreciate your insight.
    > Based on the responses I get, I am planning on writing a proposal to my
    > management to consider Python instead of Java.
    >
    > Thanks,
    > John
    >
     
    Larry Bates, May 18, 2005
    #4
  5. john67

    john67 Guest

    Would the allocation/deallocation memory usage issue be different than
    it would be with Java? Both Python and Java have automatic garbage
    collection, correct? Is Python's not as effective as Java's? I think
    the memory issues for the two languages would be similar in this area,
    but maybe I am wrong.

    John
     
    john67, May 18, 2005
    #5
  6. "keirr" <> writes:

    [snap]

    > I wouldn't, especially[1] if your thousands of business objects get
    > allocated/deallocated as the system runs. Currently python's memory
    > usage can grow rapidly (from the perspective of the o/s) when large
    > numbers of objects are repeatedly created and freed.


    Isn't it true that in recent Python releases, one may replace the
    default GC with custom one?

    > On the other hand, you could use a combination of python and C++
    > (perhaps using boost's python wrapping code). The ability to create
    > C/C++ modules is a "get out of jail free card" - if you did manage
    > to convince your company to go with python, and it fell short in
    > some area, a custom C module could be a handy trick.


    In fact, it is not a trick -- combining C with Python is fairly
    approved way of speeding up the app :)

    --
    http://www.peter.dembinski.prv.pl
     
    Peter Dembinski, May 18, 2005
    #6
  7. john67

    keirr Guest

    >> I wouldn't, especially[1] if your thousands of business objects get
    >> allocated/deallocated as the system runs. Currently python's memory
    >> usage can grow rapidly (from the perspective of the o/s) when large
    >> numbers of objects are repeatedly created and freed.


    >Isn't it true that in recent Python releases, one may replace the
    >default GC with custom one?


    Good point. Indeed, since you have the source code you can change
    anything you like. My warning was aimed at a specific situation, and
    was meant to be a 'keep this in mind' comment (I phrased it as "don't
    use python" really because I expected the majority of the responses to
    be "do use python", and thought a bit of balance would be helpful).

    As for a custom garbage collector, there has been a little discussion
    about the gc code (on this group). My feeling is, if it was easy to
    make it better someone would have done it already. Note I don't say
    it's impossible (particularly with the resources the op mentioned),
    just not straight-forward.

    Of course, I wouldn't use Java, but that's another story '-)

    Cheers,

    Keir.
     
    keirr, May 18, 2005
    #7
  8. john67

    Dave Brueck Guest

    john67 wrote:
    > The company I work for is about to embark on developing a commercial
    > application that will cost us tens-of-millions to develop.

    [snip]
    > Right now it looks like Java is the language of choice that the app
    > will be developed in. However, I have been looking and reading a lot
    > about Python recently and it seems to me that Python could handle it.
    > The big attraction to me is the developer productivity. It seems that
    > if Python can handle it, then we could gain a huge savings by using
    > Python instead of Java from a productivity standpoint alone.
    >
    > So, given the very general requirements in the first paragraph, do you
    > think that Python could handle it?


    Yes.

    Python has good database support, it works well on a wide range of platforms,
    and it's great at tying together different processes, machines, etc. - for
    example, it's fairly easy to get Python to access C code, dynamic libraries,
    system APIs, and external programs. It's easier to test than Java code as well,
    and the overal cost of change is lower.

    The concise nature of the language means that what would otherwise be a "huge"
    enterprise app in Java might just be a "large" enterprise app in Python - the
    project will not reach that unwieldy size as quickly, if ever.

    The Google archives of this group have this topic covered in various ways many
    times over - IMO Python is great for smallish apps, but its advantage over e.g.
    Java actually *increases* with the size of the app.

    If you go the Python route, two of the main obstacles will be:

    1) mindshare - getting people on board, getting them to overcome biases one way
    or another, convincing them to really take the time to come up to speed on Python.

    2) reducing the complexity of what you think you need to build. I don't know how
    to better describe this, but in Java, for example, I'd have whole bunches custom
    classes to do various tasks but the Python equivalent collapsed into a single
    module. Anyway, I predict that it'll take you some time to convince yourself
    that you simply won't need to build all of the same components as you otherwise
    would, or that they'll often be vastly simpler.

    -Dave
     
    Dave Brueck, May 18, 2005
    #8
  9. In article <>,
    john67 <> wrote:
    >The company I work for is about to embark on developing a commercial
    >application that will cost us tens-of-millions to develop. When all is
    >said and done it will have thousands of business objects/classes, some
    >of which will have hundreds-of-thousands of instances stored in a DB.
    >Our clients will probably have somewhere between 50-200 users working
    >on the app during the day, possibly in mutiple offices, and then a
    >large number of batch processes will have to run each night. It will
    >also need to have a web interface for certain users. It needs to be
    >robust, easy to maintain, and able to be customized for each client.
    >
    >Right now it looks like Java is the language of choice that the app
    >will be developed in. However, I have been looking and reading a lot
    >about Python recently and it seems to me that Python could handle it.
    >The big attraction to me is the developer productivity. It seems that
    >if Python can handle it, then we could gain a huge savings by using
    >Python instead of Java from a productivity standpoint alone.
    >
    >So, given the very general requirements in the first paragraph, do you
    >think that Python could handle it? If anyone has direct experience
    >developing large apps in Python, I would appreciate your insight.
    >Based on the responses I get, I am planning on writing a proposal to my
    >management to consider Python instead of Java.

    .
    .
    .
    My flippant response is that you'll need to choose Java if
    you want it to cost that much.

    Phaseit regularly develops applications that serve *thousands*
    of users, with database record counts in the millions, and our
    total costs are typically in the tens of thousands of dollars,
    rather than tens of millions.

    On the other hand, maybe I don't want you doing this in Python.
    Most enterprise projects fail--or at least there's plausible
    evidence to believe that most fail--and perhaps I shouldn't
    encourage Python's use in something that will fail.
     
    Cameron Laird, May 18, 2005
    #9
  10. john67

    Steve M Guest

    This thread:

    http://mail.python.org/pipermail/python-dev/2005-January/051255.html

    discusses the problem with memory allocation in CPython. Apparently
    CPython is not good at, or incapable of, releasing memory back to the
    operating system. There are ways to compensate for this. I guess the
    comment about C modules was meant as one way to do so, either by
    reducing memory requirement in the first place (C data structures are
    more compact than Python) or else by allocating and freeing memory
    wholly in the C module, which perhaps does work as expected.

    The web page for Evan Jones does not obviously indicate the status of
    the improved memory allocator patch he was working on. I wonder if it
    is coming along.

    Incidentally, does anyone know the prospects for CPython to be made
    stackless? Say in 2.5 or 2.9? Or will that always be an independent
    project?
     
    Steve M, May 18, 2005
    #10
  11. john67

    john67 Guest

    LOL! That is really funny, in a dark humor kind of way. I don't want
    the project to fail either. I am not convinced that we will succeed if
    we go the Java route. However, I am just a grunt in the chain and not
    in a position to make the decision. I hope I can have some influence
    on the decision though. I will be pleased if I can get someone to at
    least seriously consider Python.

    I understand your sentiment about not encouraging something that might
    fail to be done in Python. Hopefully it won't come to that. Thanks
    for the info on Phaseit apps.

    John
     
    john67, May 18, 2005
    #11
  12. john67

    EHP Guest

    > Python has good database support, it works well on a wide range of
    > platforms, and it's great at tying together different processes, machines,
    > etc. - for example, it's fairly easy to get Python to access C code,
    > dynamic libraries, system APIs, and external programs. It's easier to test
    > than Java code as well, and the overal cost of change is lower.


    Python has good DB support - but only simple connectors (like JDBC).
    ORM in Java (like Hibernate) are much better than ORM in Python (probably the
    best is sqlobject). How you can write huge OO app without ORM ?
    IMHO supporting libraries for bussiness apps are better in Java (don't flame
    please).

    EHP
     
    EHP, May 18, 2005
    #12
  13. Hi All--

    john67 wrote:
    >
    > LOL! That is really funny, in a dark humor kind of way. I don't want
    > the project to fail either. I am not convinced that we will succeed if
    > we go the Java route. However, I am just a grunt in the chain and not
    > in a position to make the decision. I hope I can have some influence
    > on the decision though. I will be pleased if I can get someone to at
    > least seriously consider Python.
    >
    > I understand your sentiment about not encouraging something that might
    > fail to be done in Python. Hopefully it won't come to that. Thanks
    > for the info on Phaseit apps.
    >


    What you're going to run into are two major stumbling blocks. One,
    Python's got no credibility with management types unless the
    credibility's already there. "Python? Never heard of it. Tell me
    about it. ... Oh, it's interpreted, is it? Interesting." You can
    see Python going down the sewer pipes, right on their faces. Two,
    security. "This python sounds pretty interesting. Tell me about the
    security. How can we prevent people from stealing our source code,
    which we just spent millions developing? ... Hmm, trust the developers
    out there not to peek? Oh, sure, let's use it." (True, there are ways
    around the second, but you're going to have to talk _very_ fast and have
    ALL the answers before the management type gets to his/her office and
    shuts the door in your face and on your idea.)

    Metta,
    Ivan
    ----------------------------------------------
    Ivan Van Laningham
    God N Locomotive Works
    http://www.andi-holmes.com/
    http://www.foretec.com/python/workshops/1998-11/proceedings.html
    Army Signal Corps: Cu Chi, Class of '70
    Author: Teach Yourself Python in 24 Hours
     
    Ivan Van Laningham, May 18, 2005
    #13
  14. john67

    Kay Schluehr Guest

    john67 wrote:
    > Would the allocation/deallocation memory usage issue be different

    than
    > it would be with Java? Both Python and Java have automatic garbage
    > collection, correct?


    In recent Python versions the CPython interpreter offers a
    cycle-collector which weakens the most profound counter argument
    against reference-counting techniques.

    To answer Your initial question: there is probably no technical reason
    against Python as a language or the CPython runtime. Both are very
    stable and mature. Extension and optimization techniques are well
    understood. The library support is great. I would be more concerned
    about the development strategy. Be aware that Python is weak in
    defining constraints on interfaces even more than Java ( Javas static
    type system makes a bit easier what doesn't mean that static typing is
    a really adequate solution for it at all ). There are almost no
    declarative elements in the language. You have somehow to think about
    communicating contracts between different developers across the team.
    Elements of lightweight methodologys like continous integration and
    early unit-testing are mandatory, not optional. Be also aware that
    there are no IDEs / GUI-builders and UML-designers which are comparable
    to those for Java or dotNET. You won't come as close to a group
    consensus as if You would stick to VisualStudio7 for C#, or
    IDEA/Eclipse for Java.

    Regards,
    Kay
     
    Kay Schluehr, May 18, 2005
    #14
  15. john67

    Dave Brueck Guest

    EHP wrote:
    >>Python has good database support, it works well on a wide range of
    >>platforms, and it's great at tying together different processes, machines,
    >>etc. - for example, it's fairly easy to get Python to access C code,
    >>dynamic libraries, system APIs, and external programs. It's easier to test
    >>than Java code as well, and the overal cost of change is lower.

    >
    >
    > Python has good DB support - but only simple connectors (like JDBC).
    > ORM in Java (like Hibernate) are much better than ORM in Python (probably the
    > best is sqlobject). How you can write huge OO app without ORM ?


    I don't doubt that Hibernate is more mature, but having not used it, I can't
    really say how much better/worse it is than, say, sqlobject for Python. But, as
    you pointed out, there *are* ORM libraries for Python.

    FWIW, the only times I've been involved in database-centric projects with a cost
    in the "tens of millions", the database team steered clear of any sort of any
    automatic ORM layer altogether. Depending on your application, an ORM can fall
    into the same category as, say, EJB - a nifty idea that looks great on paper but
    can cause more problems than it solves.

    Just because the app itself is very OO, it doesn't always follow that the
    database level needs to be - there are lots and lots of problems for which
    normal RDBMS tables & joins are a pretty darn good fit. (I'm not trying to
    discount ORM layers or even object databases, just disagreeing with the notion
    that huge OO apps automatically require ORM)

    Have fun,
    -Dave
     
    Dave Brueck, May 18, 2005
    #15
  16. On Wed, May 18, 2005 at 08:37:00AM -0700, john67 wrote:
    > The company I work for is about to embark on developing a commercial
    > application that will cost us tens-of-millions to develop. When all is
    > said and done it will have thousands of business objects/classes, some
    > of which will have hundreds-of-thousands of instances stored in a DB.
    > Our clients will probably have somewhere between 50-200 users working
    > on the app during the day, possibly in mutiple offices, and then a
    > large number of batch processes will have to run each night. It will
    > also need to have a web interface for certain users. It needs to be
    > robust, easy to maintain, and able to be customized for each client.

    [snip]

    The technical specifications seem OK, these aren't language constraints
    in python more than any other language. The people constraints...

    Tens of millions to develop? Whoa - it is easy for me to predict absolute
    and complete failure right off the bat no matter the language (but only
    after incurring cost overruns). If your head is anywhere near the chopping
    block I suggest proffering a "proof of concept" version first. If you
    can't do a first version in six months with a team of six people it is a sign
    that you don't really know what you want. The final version might be
    bigger and have more features - but 10M of features is pie in the sky stuff.
    Everyone involved must have wild dreams about how awesome the final version
    will be, and all those dreams diverge.

    Ten million is ten guys writing for two years (top notch at US prices) to come
    up with a product from sratch. Tens of millions is? Best of luck.

    -jackdied
     
    Jack Diederich, May 18, 2005
    #16
  17. john67

    Andrew Dalke Guest

    Ivan Van Laningham wrote:
    > ... Oh, it's interpreted, is it? Interesting." You can
    > see Python going down the sewer pipes, right on their faces.


    Nahh, the right answer is "It's byte-compiled, just like Java."

    Andrew
     
    Andrew Dalke, May 18, 2005
    #17
  18. "Ivan Van Laningham" wrote:

    > What you're going to run into are two major stumbling blocks. One,
    > Python's got no credibility with management types unless the
    > credibility's already there. "Python? Never heard of it. Tell me
    > about it. ... Oh, it's interpreted, is it? Interesting." You can
    > see Python going down the sewer pipes, right on their faces. Two,
    > security. "This python sounds pretty interesting. Tell me about the
    > security. How can we prevent people from stealing our source code,
    > which we just spent millions developing? ... Hmm, trust the

    developers
    > out there not to peek? Oh, sure, let's use it." (True, there are

    ways
    > around the second, but you're going to have to talk _very_ fast and

    have
    > ALL the answers before the management type gets to his/her office and
    > shuts the door in your face and on your idea.)


    Another issue that hasn't been mentioned so far is staffing. You can
    find java programmers and software engineers a dime dozen, but for
    python it will be harder. The alternative of teaching python to a team
    for a week or a month is IMO far too risky when you're talking about
    millions; even if they are able to read or code fast in a new language,
    they won't have grasped the idioms and the good programming techniques,
    so they will be essentially mentally translating from java/c++/c#,
    whatever their programming mother tongue is. On the bright side, in
    python you won't have to try as hard to pick the few good programmers
    among a sea of average/inexperienced/clueless ones that you'll have for
    java.

    Personally, although I find nothing comes close to the clarity and
    flexibility that python offers, in this case I would go with java, if
    for nothing else, to be on the safe side. Think about it: if the
    project fails (and that's quite likely for huge projects, not dark
    humour) and you've done it in python, everyone will blame python and
    you who chose it; java OTOH belongs in the realm of "standard business
    practices", so in a way "it's ok" to screw up, you did what everybody
    else does. Yet another failed java enterprise project, nothing to see
    here.

    An idea that perhaps takes the best of both worlds is use java for the
    high level architecture and static type interfaces, and write the bulk
    of the implementation in jython. PSF has awarded a grant to make jython
    catch up with cpython, and that's good news for making the transition
    from java to python smoother to a large audience. Others may have more
    to say on the pros and cons of going with java/jython instead of
    cpython, but it seems a good compromise to me.

    George
     
    George Sakkis, May 18, 2005
    #18
  19. john67

    Guest

    john67 wrote:
    > [...] that will cost us tens-of-millions to develop.
    > [...]
    > Right now it looks like Java is the language of choice that the app
    > will be developed in. However, I have been looking and reading a lot
    > about Python recently [...]


    Ignoring the technical aspects: if you have good group of people that
    already know Java, and don't know Python, then use Java for this huge
    project. Do a smaller project in Python first for the learning curve,
    to build expertise and confidence, etc. People are more important than
    programming language.

    Brian.
     
    , May 18, 2005
    #19
  20. john67

    Dave Brueck Guest

    George Sakkis wrote:
    > Personally, although I find nothing comes close to the clarity and
    > flexibility that python offers, in this case I would go with java, if
    > for nothing else, to be on the safe side. Think about it: if the
    > project fails (and that's quite likely for huge projects, not dark
    > humour) and you've done it in python, everyone will blame python and
    > you who chose it; java OTOH belongs in the realm of "standard business
    > practices", so in a way "it's ok" to screw up, you did what everybody
    > else does. Yet another failed java enterprise project, nothing to see
    > here.


    lol - I got a kick out of this post and I agree with it: *if* you're pretty sure
    the project will ultimately fail, and you're planning for the best way to point
    fingers and avoid the ax when it happens, then by all means, Java is the right
    way to go. Java is the new "nobody ever got fired for buying IBM".

    -Dave
     
    Dave Brueck, May 18, 2005
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    3
    Views:
    583
  2. Replies:
    3
    Views:
    415
    Tom Dyess
    Jan 20, 2005
  3. Replies:
    2
    Views:
    605
    Tom Dyess
    Jan 20, 2005
  4. Replies:
    3
    Views:
    554
  5. Asaf Hayman
    Replies:
    4
    Views:
    330
    Jason Scheirer
    Nov 18, 2008
Loading...

Share This Page