client/server design and advice

Discussion in 'Python' started by TonyM, Dec 1, 2006.

  1. TonyM

    TonyM Guest

    I recently completed the general guidelines for a future project that I
    would like to start developing...but I've sort of hit a wall with
    respect to how to design it. In short, I want to run through
    approximately 5gigs of financial data, all of which is stored in a
    large number of text files. Now as far as formatting and data
    integrity...I would go through and ensure that each file had the
    required setup so thats not really the issue. The problem I am having
    is with respect to speed.

    The languages I knew the best when coming into this project includes
    c++ and php. However, I then thought about how long it would take one
    PC to iterate through everything and figured it would probably take a
    significant amount of time. As such, I started looking into various
    languages and python caught my interest the most due to its power and
    what seems to be ease of use. I was going to initially just use python
    as a means of creating various indicators (i.e. calculations that would
    be performed on the data in the file)...however I am leaning towards
    moving to python entirely mostly due to its gui support.

    First off, i was wondering if this is a reasonable setup: The entire
    process would involve a server which manages which pc is processing
    which set of data (which may be a given text file or the like), and a
    client application which i would run on a few pc's locally when they
    aren't in use. I would have a database (sqlite) holding all calculated
    data of significance. Each client will basically login/connect with
    the server, request a time interval (i.e. does anything need processed?
    if so what data should i look at), and then it would update its status
    with the server which would place a lock on that data set.

    One thing i was wondering is if it would be worth it to use c++ for the
    actual iteration through the text file or should i simply use python?
    While i'm sure that c++ would be faster i am not entirely sure its
    worth the headache if its not going to save me significant processing
    time. Another thing is...if i was going to work with python instead of
    c++, would it be worth it to import all of the data into an sqlite
    database before hand (for speed issues)?

    Lastly, as far as the networking goes, i have seen posts and such about
    something called Pyro (http://pyro.sourceforge.net) and wondered if
    that was worth looking into for the client/server interaction.

    I apologize if any of these questions are more lower level, this is
    simply the first client/server application ive created and am doing so
    in a language ive never used before ;)

    Thanks for the help
    -Tony
     
    TonyM, Dec 1, 2006
    #1
    1. Advertising

  2. > First off, i was wondering if this is a reasonable setup: The entire
    > process would involve a server which manages which pc is processing
    > which set of data (which may be a given text file or the like), and a
    > client application which i would run on a few pc's locally when they
    > aren't in use. I would have a database (sqlite) holding all calculated
    > data of significance. Each client will basically login/connect with
    > the server, request a time interval (i.e. does anything need processed?
    > if so what data should i look at), and then it would update its status
    > with the server which would place a lock on that data set.


    Don't use sqlite, use a "real" RDBMS. sqlite is cool, but not really suited
    for large amounts of data, and the concurrent access aspects that are dealt
    with with an RDBMS for free are not to be underestimated.

    > One thing i was wondering is if it would be worth it to use c++ for the
    > actual iteration through the text file or should i simply use python?
    > While i'm sure that c++ would be faster i am not entirely sure its
    > worth the headache if its not going to save me significant processing
    > time. Another thing is...if i was going to work with python instead of
    > c++, would it be worth it to import all of the data into an sqlite
    > database before hand (for speed issues)?


    I'd be putting them in the DB, yes.

    > Lastly, as far as the networking goes, i have seen posts and such about
    > something called Pyro (http://pyro.sourceforge.net) and wondered if
    > that was worth looking into for the client/server interaction.


    Pyro rocks for that.

    Diez
     
    Diez B. Roggisch, Dec 1, 2006
    #2
    1. Advertising

  3. TonyM

    TonyM Guest

    > Don't use sqlite, use a "real" RDBMS. sqlite is cool, but not really suited
    > for large amounts of data, and the concurrent access aspects that are dealt
    > with with an RDBMS for free are not to be underestimated.


    Would PostgreSQL be suitable in this situation? I hadn't even thought
    about the possible problems that could arise with concurrency but i do
    recall it being an issue the last time i worked with sqlite. I have
    also looked into mysql given my extensive experience with it...however
    postgresql seems to be faster from what ive read. Either way i'll work
    on writing something to convert and insert the data so that it can
    process while im working on the gui and client/server apps.

    > Pyro rocks for that.


    Awesome, ill look into it in greater detail and will most likely use
    it. Given what ive seen so far it looks like it will make the
    client/server interface fairly easy to write.

    Now...if only i could master python gui programming and development ;)
    I'm not entirely sure which gui lib im going to end up using, but as of
    now im leaning more towards tkinter as i know it will work where i need
    and it seems to be one of the more documented. Ive looked at used
    wxpython a little but had trouble figure out a few minor things while
    playing around with it initially. I've also thought about pygtk
    although I haven't taken the time to play with it quite yet as i
    assumed it was primarily for linux (id be running the majority of these
    on windows pcs).

    Thanks for the suggestions :)
    Tony
     
    TonyM, Dec 1, 2006
    #3
  4. TonyM wrote:

    > Lastly, as far as the networking goes, i have seen posts and such about
    > something called Pyro (http://pyro.sourceforge.net) and wondered if
    > that was worth looking into for the client/server interaction.


    I'm currently busy with a new version of Pyro (3.6) and it already
    includes a new 'distributed computing' example, where there is
    a single dispatcher service and one or more 'worker' clients.
    The clients request work 'packets' from the dispatcher and
    process them in parallel.
    Maybe this is a good starting point of your system?
    Current code is available from Pyro's CVS repository.

    --Irmen
     
    Irmen de Jong, Dec 1, 2006
    #4
  5. TonyM

    John Henry Guest

    TonyM wrote:
    <snip>

    > > Pyro rocks for that.

    >
    > Awesome, ill look into it in greater detail and will most likely use
    > it. Given what ive seen so far it looks like it will make the
    > client/server interface fairly easy to write.
    >


    Correction: not "fairly easy" - make that "incredibly easy". Even
    Micky likes it. :=)

    > Now...if only i could master python gui programming and development ;)
    > I'm not entirely sure which gui lib im going to end up using, but as of
    > now im leaning more towards tkinter as i know it will work where i need
    > and it seems to be one of the more documented. Ive looked at used
    > wxpython a little but had trouble figure out a few minor things while
    > playing around with it initially. I've also thought about pygtk
    > although I haven't taken the time to play with it quite yet as i
    > assumed it was primarily for linux (id be running the majority of these
    > on windows pcs).
    >


    You would short change yourself if you don't check out the other
    packages such as Pythoncard, and Dabo.

    The other thing I recommend for large scale applications:

    http://www-128.ibm.com/developerworks/library/l-pythrd.html


    > Thanks for the suggestions :)
    > Tony
     
    John Henry, Dec 1, 2006
    #5
  6. TonyM

    Peter Decker Guest

    On 1 Dec 2006 10:37:39 -0800, John Henry <> wrote:

    > > Now...if only i could master python gui programming and development ;)


    > You would short change yourself if you don't check out the other
    > packages such as Pythoncard, and Dabo.


    FWIW, Dabo has all of the database connectivity stuff built-in. With
    PythonCard, you have to roll your own.

    --

    # p.d.
     
    Peter Decker, Dec 1, 2006
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Junpei
    Replies:
    1
    Views:
    497
    Jim Gibson
    May 27, 2004
  2. Dominic
    Replies:
    1
    Views:
    375
    AdamM
    Dec 11, 2003
  3. -
    Replies:
    2
    Views:
    445
    Alan Krueger
    Jul 29, 2005
  4. bruce
    Replies:
    3
    Views:
    289
    Irmen de Jong
    Dec 2, 2006
  5. Mark
    Replies:
    11
    Views:
    469
    Stuart MacMartin
    Sep 7, 2005
Loading...

Share This Page