Converting existing module/objects to threads

Discussion in 'Python' started by jdlists@gmail.com, Oct 19, 2006.

  1. Guest

    I have inheirted some existing code, that i will explain in a moment,
    have needed to extend and ultimately should be able to run in threads.
    I've done a bunch of work with python but very little with threads and
    am looking for some pointers on how to implement, and if the lower
    level modules/objects need to be rewritten to use threading.local for
    all local variables.

    I have a module that communicates with a hardware device, which reads
    data off of sensors, that can only talk with one controller at a time.
    The controller (my module) needs to (in its simplest form) init,
    configure the device, request data, and write out xml, sleep, repeat.

    The new request is that the device needs to be queried until a
    condition is true, and then start requesting data. So an instance of a
    controller needs to be deadicated to a hardware device forever, or
    until the program ends....which ever comes first.

    This currently works in a non-threaded version, but only for one device
    at a time, there is a need to create a single windows(yeach) service
    that talks to many of these devices at once. I don't need worker
    threads that handle seperate portions of the entire job, i need a
    single application to spawn multiple processes to run through the
    entire communication from configure to report, sleep until the next
    interval time and run again. The communication could last from 1
    minute to 10 minutes before it ends.


    Here is the code layout in pseudocode.

    module.Object - controller.Main - handles all socket communications

    class subcontroller(controller.Main):
    def __init__(self,id,configurationFile):
    controller.Main.__init__(self)
    // instantiate variables and local objects that handle
    configuration, logic and data output

    def configure(self,configurationFile):
    //read configurationFile and configure device

    def process(self):
    while 1:
    //based on configuration file, query the device until condition
    is true and then write xml, sleep until time to repeat and run again.

    within controller there are 5 objects and subcontroller is a sinlge
    object that loads other objects from the inherited controller.System

    I'm trying to figure out how difficult it is going to be to convert
    this to a threaded application. The original controller.Main is built
    to talk to devices in series, never in parallel. so no objects are
    considered to be thread safe, but no instance of any of the objects
    should need to share resources with any other instance of teh same
    object. they would all have unique configuration files and talk to
    devices on unique ip/ports.

    on a unix system, forking,while potentially not optimal, would be a
    fine solution, unfortunantely this needs to run on windows.

    I know i have left out many details, but hopefully this is enough to at
    least enable some kind soles to lend an opinnion or two.

    many thanks
    jd
     
    , Oct 19, 2006
    #1
    1. Advertising

  2. At Wednesday 18/10/2006 22:02, wrote:

    >This currently works in a non-threaded version, but only for one device
    >at a time, there is a need to create a single windows(yeach) service
    >that talks to many of these devices at once. I don't need worker
    >threads that handle seperate portions of the entire job, i need a
    >single application to spawn multiple processes to run through the
    >entire communication from configure to report, sleep until the next
    >interval time and run again. The communication could last from 1
    >minute to 10 minutes before it ends.


    Consider using the asyncore module instead of threads.


    --
    Gabriel Genellina
    Softlab SRL





    __________________________________________________
    Preguntá. Respondé. Descubrí.
    Todo lo que querías saber, y lo que ni imaginabas,
    está en Yahoo! Respuestas (Beta).
    ¡Probalo ya!
    http://www.yahoo.com.ar/respuestas
     
    Gabriel Genellina, Oct 19, 2006
    #2
    1. Advertising

  3. Ben Finney Guest

    Corner cases (was: Converting existing module/objects to threads)

    "" <> writes:

    > So an instance of a controller needs to be deadicated to a hardware
    > device forever, or until the program ends....which ever comes first.


    I hope you've got tests in place for both of those conditions :)

    --
    \ "How many people here have telekenetic powers? Raise my hand." |
    `\ -- Emo Philips |
    _o__) |
    Ben Finney
     
    Ben Finney, Oct 19, 2006
    #3
  4. martdi Guest

    wrote:
    > I have inheirted some existing code, that i will explain in a moment,
    > have needed to extend and ultimately should be able to run in threads.
    > I've done a bunch of work with python but very little with threads and
    > am looking for some pointers on how to implement, and if the lower
    > level modules/objects need to be rewritten to use threading.local for
    > all local variables.
    >
    > I have a module that communicates with a hardware device, which reads
    > data off of sensors, that can only talk with one controller at a time.
    > The controller (my module) needs to (in its simplest form) init,
    > configure the device, request data, and write out xml, sleep, repeat.
    >
    > The new request is that the device needs to be queried until a
    > condition is true, and then start requesting data. So an instance of a
    > controller needs to be deadicated to a hardware device forever, or
    > until the program ends....which ever comes first.
    >
    > This currently works in a non-threaded version, but only for one device
    > at a time, there is a need to create a single windows(yeach) service
    > that talks to many of these devices at once. I don't need worker
    > threads that handle seperate portions of the entire job, i need a
    > single application to spawn multiple processes to run through the
    > entire communication from configure to report, sleep until the next
    > interval time and run again. The communication could last from 1
    > minute to 10 minutes before it ends.
    >
    >
    > Here is the code layout in pseudocode.
    >
    > module.Object - controller.Main - handles all socket communications
    >
    > class subcontroller(controller.Main):
    > def __init__(self,id,configurationFile):
    > controller.Main.__init__(self)
    > // instantiate variables and local objects that handle
    > configuration, logic and data output
    >
    > def configure(self,configurationFile):
    > //read configurationFile and configure device
    >
    > def process(self):
    > while 1:
    > //based on configuration file, query the device until condition
    > is true and then write xml, sleep until time to repeat and run again.
    >
    > within controller there are 5 objects and subcontroller is a sinlge
    > object that loads other objects from the inherited controller.System
    >
    > I'm trying to figure out how difficult it is going to be to convert
    > this to a threaded application. The original controller.Main is built
    > to talk to devices in series, never in parallel. so no objects are
    > considered to be thread safe, but no instance of any of the objects
    > should need to share resources with any other instance of teh same
    > object. they would all have unique configuration files and talk to
    > devices on unique ip/ports.
    >
    > on a unix system, forking,while potentially not optimal, would be a
    > fine solution, unfortunantely this needs to run on windows.
    >
    > I know i have left out many details, but hopefully this is enough to at
    > least enable some kind soles to lend an opinnion or two.
    >
    > many thanks
    > jd


    Taking a look at asyncore could be worthwhile, but if you want to
    implement it with threads, you may be able to do it this way:

    In your main file, from where you start the program, let's call it
    main:

    main(self)
    Load Required configuration
    spawn threads (1 for each controller)
    define a queue object from module queue.queue used for communication
    with threads
    enter an infinite loop that checks for the conditions
    once conditions are met, notify the proper thread


    class ControllerThread(threading.Thread):
    def __init__(self):
    define a queue here, to process messages from the main
    call threading.Thread.__init__(self)

    define method to load config for thread objects
    (you might want to pass an argument to init to load your configs
    from a file)
    (you might also want to pass the queue of the main program to the
    thread to send it messages)
    define methods to post messages to the queue like read, send to the
    machine, stop, ...

    define the run method that is what will be called when you start
    your thread.
    this method should enter an infinite loop that will check if
    something has to be done (check in the queue).





    hope this might help you
    good luck
     
    martdi, Oct 19, 2006
    #4
  5. Guest

    Gabriel Genellina wrote:
    > At Wednesday 18/10/2006 22:02, wrote:
    >
    > >This currently works in a non-threaded version, but only for one device
    > >at a time, there is a need to create a single windows(yeach) service
    > >that talks to many of these devices at once. I don't need worker
    > >threads that handle seperate portions of the entire job, i need a
    > >single application to spawn multiple processes to run through the
    > >entire communication from configure to report, sleep until the next
    > >interval time and run again. The communication could last from 1
    > >minute to 10 minutes before it ends.

    >
    > Consider using the asyncore module instead of threads.
    >
    >
    > --
    > Gabriel Genellina
    > Softlab SRL


    I think that is a good point and I am considering using
    asyncore/asynchat... i'm a little confused as to how i can make this
    model work. There is no server communication without connection from
    the client (me), which happens on intervals, not when data is available
    on a socket or when the socket is available to be written, which is
    always. Basically i need to determine how to trigger the asynchat
    process based on time. in another application that i write, i'm the
    server and the chat process happens every time the client wakes
    up...easy and perfect for asyncore

    That is a solution i'd like to persue, but am having a hard time
    getting my head around that as well.



    >
    >
    >
    >
    >
    > __________________________________________________
    > Preguntá. Respondé. Descubrí.
    > Todo lo que querías saber, y lo que ni imaginabas,
    > está en Yahoo! Respuestas (Beta).
    > ¡Probalo ya!
    > http://www.yahoo.com.ar/respuestas
     
    , Oct 19, 2006
    #5
  6. Guest

    martdi wrote:
    > wrote:
    > > I have inheirted some existing code, that i will explain in a moment,
    > > have needed to extend and ultimately should be able to run in threads.
    > > I've done a bunch of work with python but very little with threads and
    > > am looking for some pointers on how to implement, and if the lower
    > > level modules/objects need to be rewritten to use threading.local for
    > > all local variables.
    > >
    > > I have a module that communicates with a hardware device, which reads
    > > data off of sensors, that can only talk with one controller at a time.
    > > The controller (my module) needs to (in its simplest form) init,
    > > configure the device, request data, and write out xml, sleep, repeat.
    > >
    > > The new request is that the device needs to be queried until a
    > > condition is true, and then start requesting data. So an instance of a
    > > controller needs to be deadicated to a hardware device forever, or
    > > until the program ends....which ever comes first.
    > >
    > > This currently works in a non-threaded version, but only for one device
    > > at a time, there is a need to create a single windows(yeach) service
    > > that talks to many of these devices at once. I don't need worker
    > > threads that handle seperate portions of the entire job, i need a
    > > single application to spawn multiple processes to run through the
    > > entire communication from configure to report, sleep until the next
    > > interval time and run again. The communication could last from 1
    > > minute to 10 minutes before it ends.
    > >
    > >
    > > Here is the code layout in pseudocode.
    > >
    > > module.Object - controller.Main - handles all socket communications
    > >
    > > class subcontroller(controller.Main):
    > > def __init__(self,id,configurationFile):
    > > controller.Main.__init__(self)
    > > // instantiate variables and local objects that handle
    > > configuration, logic and data output
    > >
    > > def configure(self,configurationFile):
    > > //read configurationFile and configure device
    > >
    > > def process(self):
    > > while 1:
    > > //based on configuration file, query the device until condition
    > > is true and then write xml, sleep until time to repeat and run again.
    > >
    > > within controller there are 5 objects and subcontroller is a sinlge
    > > object that loads other objects from the inherited controller.System
    > >
    > > I'm trying to figure out how difficult it is going to be to convert
    > > this to a threaded application. The original controller.Main is built
    > > to talk to devices in series, never in parallel. so no objects are
    > > considered to be thread safe, but no instance of any of the objects
    > > should need to share resources with any other instance of teh same
    > > object. they would all have unique configuration files and talk to
    > > devices on unique ip/ports.
    > >
    > > on a unix system, forking,while potentially not optimal, would be a
    > > fine solution, unfortunantely this needs to run on windows.
    > >
    > > I know i have left out many details, but hopefully this is enough to at
    > > least enable some kind soles to lend an opinnion or two.
    > >
    > > many thanks
    > > jd

    >
    > Taking a look at asyncore could be worthwhile, but if you want to
    > implement it with threads, you may be able to do it this way:
    >
    > In your main file, from where you start the program, let's call it
    > main:
    >
    > main(self)
    > Load Required configuration
    > spawn threads (1 for each controller)
    > define a queue object from module queue.queue used for communication
    > with threads
    > enter an infinite loop that checks for the conditions
    > once conditions are met, notify the proper thread
    >
    >
    > class ControllerThread(threading.Thread):
    > def __init__(self):
    > define a queue here, to process messages from the main
    > call threading.Thread.__init__(self)
    >
    > define method to load config for thread objects
    > (you might want to pass an argument to init to load your configs
    > from a file)
    > (you might also want to pass the queue of the main program to the
    > thread to send it messages)
    > define methods to post messages to the queue like read, send to the
    > machine, stop, ...
    >
    > define the run method that is what will be called when you start
    > your thread.
    > this method should enter an infinite loop that will check if
    > something has to be done (check in the queue).
    >
    >
    >
    >
    >
    > hope this might help you
    > good luck


    thanks for the comments. the ControllerThread already extends a class.
    does this cause problems with classes that must extent
    threading.Thread. Normally it should not matter, but with threads i'm
    unsure. Should i just instantiate the object that i'm normally
    extending in the ControllerThread.__init__, and call it from the
    self.classthatusedtoextend.method(), or does it not matter.

    again, thanks.
     
    , Oct 19, 2006
    #6
  7. martdi Guest

    I am not sure if I understand you question well, but:

    in the __init__ of the thread subclass, you can instantiate an object
    of the class that makes the work

    or

    ControllerThread could extend both classes and i don't think there
    would be a problem.



    Problems in multithreading usually happen when many threads try to
    access the same ressource at the same time or that one thread waits for
    an other thread to finish, and that other thread waits for the first
    one to finish. The Queue module is threadsafe and it uses locking to
    prevent multiple threads to access it at the same time. Using Queue is
    a lot easier than having to manage locks by yourself.

    As long as you do not share data between your classes like static
    members, database connections, or I/0 on the same ressource, you
    probably won't encounter problems.

    wrote:
    > martdi wrote:
    > > wrote:
    > > > I have inheirted some existing code, that i will explain in a moment,
    > > > have needed to extend and ultimately should be able to run in threads.
    > > > I've done a bunch of work with python but very little with threads and
    > > > am looking for some pointers on how to implement, and if the lower
    > > > level modules/objects need to be rewritten to use threading.local for
    > > > all local variables.
    > > >
    > > > I have a module that communicates with a hardware device, which reads
    > > > data off of sensors, that can only talk with one controller at a time.
    > > > The controller (my module) needs to (in its simplest form) init,
    > > > configure the device, request data, and write out xml, sleep, repeat.
    > > >
    > > > The new request is that the device needs to be queried until a
    > > > condition is true, and then start requesting data. So an instance of a
    > > > controller needs to be deadicated to a hardware device forever, or
    > > > until the program ends....which ever comes first.
    > > >
    > > > This currently works in a non-threaded version, but only for one device
    > > > at a time, there is a need to create a single windows(yeach) service
    > > > that talks to many of these devices at once. I don't need worker
    > > > threads that handle seperate portions of the entire job, i need a
    > > > single application to spawn multiple processes to run through the
    > > > entire communication from configure to report, sleep until the next
    > > > interval time and run again. The communication could last from 1
    > > > minute to 10 minutes before it ends.
    > > >
    > > >
    > > > Here is the code layout in pseudocode.
    > > >
    > > > module.Object - controller.Main - handles all socket communications
    > > >
    > > > class subcontroller(controller.Main):
    > > > def __init__(self,id,configurationFile):
    > > > controller.Main.__init__(self)
    > > > // instantiate variables and local objects that handle
    > > > configuration, logic and data output
    > > >
    > > > def configure(self,configurationFile):
    > > > //read configurationFile and configure device
    > > >
    > > > def process(self):
    > > > while 1:
    > > > //based on configuration file, query the device until condition
    > > > is true and then write xml, sleep until time to repeat and run again.
    > > >
    > > > within controller there are 5 objects and subcontroller is a sinlge
    > > > object that loads other objects from the inherited controller.System
    > > >
    > > > I'm trying to figure out how difficult it is going to be to convert
    > > > this to a threaded application. The original controller.Main is built
    > > > to talk to devices in series, never in parallel. so no objects are
    > > > considered to be thread safe, but no instance of any of the objects
    > > > should need to share resources with any other instance of teh same
    > > > object. they would all have unique configuration files and talk to
    > > > devices on unique ip/ports.
    > > >
    > > > on a unix system, forking,while potentially not optimal, would be a
    > > > fine solution, unfortunantely this needs to run on windows.
    > > >
    > > > I know i have left out many details, but hopefully this is enough to at
    > > > least enable some kind soles to lend an opinnion or two.
    > > >
    > > > many thanks
    > > > jd

    > >
    > > Taking a look at asyncore could be worthwhile, but if you want to
    > > implement it with threads, you may be able to do it this way:
    > >
    > > In your main file, from where you start the program, let's call it
    > > main:
    > >
    > > main(self)
    > > Load Required configuration
    > > spawn threads (1 for each controller)
    > > define a queue object from module queue.queue used for communication
    > > with threads
    > > enter an infinite loop that checks for the conditions
    > > once conditions are met, notify the proper thread
    > >
    > >
    > > class ControllerThread(threading.Thread):
    > > def __init__(self):
    > > define a queue here, to process messages from the main
    > > call threading.Thread.__init__(self)
    > >
    > > define method to load config for thread objects
    > > (you might want to pass an argument to init to load your configs
    > > from a file)
    > > (you might also want to pass the queue of the main program to the
    > > thread to send it messages)
    > > define methods to post messages to the queue like read, send to the
    > > machine, stop, ...
    > >
    > > define the run method that is what will be called when you start
    > > your thread.
    > > this method should enter an infinite loop that will check if
    > > something has to be done (check in the queue).
    > >
    > >
    > >
    > >
    > >
    > > hope this might help you
    > > good luck

    >
    > thanks for the comments. the ControllerThread already extends a class.
    > does this cause problems with classes that must extent
    > threading.Thread. Normally it should not matter, but with threads i'm
    > unsure. Should i just instantiate the object that i'm normally
    > extending in the ControllerThread.__init__, and call it from the
    > self.classthatusedtoextend.method(), or does it not matter.
    >
    > again, thanks.
     
    martdi, Oct 19, 2006
    #7
  8. John Henry Guest

    Making your code run in thread mode isn't the hard part. Just add
    this:

    import threading

    class subcontrollerThread(threading.Thread, subcontroller):
    def __init__(self,id,configurationFile):
    threading.Thread.__init__(self)
    subcontroller.__init__(self,id,configurationFile)
    def run(self):
    self.process()

    threads=[]
    # Say we have 5 of the subprocesses
    for iThread in range(5):
    th=subcontrollerThread(iThread,configurationFile)
    threads.append(th)
    th.start()

    ....main thread do whatever...


    However, you have to make sure the code inside subcontroller is thread
    safe. That's a topic in itself.


    wrote:
    > I have inheirted some existing code, that i will explain in a moment,
    > have needed to extend and ultimately should be able to run in threads.
    > I've done a bunch of work with python but very little with threads and
    > am looking for some pointers on how to implement, and if the lower
    > level modules/objects need to be rewritten to use threading.local for
    > all local variables.
    >
    > I have a module that communicates with a hardware device, which reads
    > data off of sensors, that can only talk with one controller at a time.
    > The controller (my module) needs to (in its simplest form) init,
    > configure the device, request data, and write out xml, sleep, repeat.
    >
    > The new request is that the device needs to be queried until a
    > condition is true, and then start requesting data. So an instance of a
    > controller needs to be deadicated to a hardware device forever, or
    > until the program ends....which ever comes first.
    >
    > This currently works in a non-threaded version, but only for one device
    > at a time, there is a need to create a single windows(yeach) service
    > that talks to many of these devices at once. I don't need worker
    > threads that handle seperate portions of the entire job, i need a
    > single application to spawn multiple processes to run through the
    > entire communication from configure to report, sleep until the next
    > interval time and run again. The communication could last from 1
    > minute to 10 minutes before it ends.
    >
    >
    > Here is the code layout in pseudocode.
    >
    > module.Object - controller.Main - handles all socket communications
    >
    > class subcontroller(controller.Main):
    > def __init__(self,id,configurationFile):
    > controller.Main.__init__(self)
    > // instantiate variables and local objects that handle
    > configuration, logic and data output
    >
    > def configure(self,configurationFile):
    > //read configurationFile and configure device
    >
    > def process(self):
    > while 1:
    > //based on configuration file, query the device until condition
    > is true and then write xml, sleep until time to repeat and run again.
    >
    > within controller there are 5 objects and subcontroller is a sinlge
    > object that loads other objects from the inherited controller.System
    >
    > I'm trying to figure out how difficult it is going to be to convert
    > this to a threaded application. The original controller.Main is built
    > to talk to devices in series, never in parallel. so no objects are
    > considered to be thread safe, but no instance of any of the objects
    > should need to share resources with any other instance of teh same
    > object. they would all have unique configuration files and talk to
    > devices on unique ip/ports.
    >
    > on a unix system, forking,while potentially not optimal, would be a
    > fine solution, unfortunantely this needs to run on windows.
    >
    > I know i have left out many details, but hopefully this is enough to at
    > least enable some kind soles to lend an opinnion or two.
    >
    > many thanks
    > jd
     
    John Henry, Oct 19, 2006
    #8
  9. At Thursday 19/10/2006 00:01, wrote:

    > > Consider using the asyncore module instead of threads.

    >
    >I think that is a good point and I am considering using
    >asyncore/asynchat... i'm a little confused as to how i can make this
    >model work. There is no server communication without connection from
    >the client (me), which happens on intervals, not when data is available
    >on a socket or when the socket is available to be written, which is
    >always. Basically i need to determine how to trigger the asynchat
    >process based on time. in another application that i write, i'm the
    >server and the chat process happens every time the client wakes
    >up...easy and perfect for asyncore
    >
    >That is a solution i'd like to persue, but am having a hard time
    >getting my head around that as well.


    You have to write your own dispatcher (inheriting from async_chat) as
    any other protocol. You can call asyncore.loop whith count=1 (or 10,
    but not None, so it returns after a few iterations) inside your *own*
    loop. Inside your loop, when time comes, call
    your_dispatcher.push(data) so the channel gets data to be sent.
    Override collect_incoming_data() to get the response.
    You can keep your pending requests in a priority queue (sorted by
    time) and check the current time against the top element's time.

    Maybe you could successfully implement your application using threads
    - if none uses global variables, and no thread waits for another, it
    may be safe. But I prefer to avoid threads whenever possible, at
    least because debugging the application becomes harder.


    --
    Gabriel Genellina
    Softlab SRL





    __________________________________________________
    Preguntá. Respondé. Descubrí.
    Todo lo que querías saber, y lo que ni imaginabas,
    está en Yahoo! Respuestas (Beta).
    ¡Probalo ya!
    http://www.yahoo.com.ar/respuestas
     
    Gabriel Genellina, Oct 19, 2006
    #9
  10. <> wrote:
    > I have inheirted some existing code, that i will explain in a moment,
    > have needed to extend and ultimately should be able to run in threads.
    > I've done a bunch of work with python but very little with threads and
    > am looking for some pointers on how to implement, and if the lower
    > level modules/objects need to be rewritten to use threading.local for
    > all local variables.
    >
    > I have a module that communicates with a hardware device, which reads
    > data off of sensors, that can only talk with one controller at a time.
    > The controller (my module) needs to (in its simplest form) init,
    > configure the device, request data, and write out xml, sleep, repeat.
    >
    > The new request is that the device needs to be queried until a
    > condition is true, and then start requesting data. So an instance of a
    > controller needs to be deadicated to a hardware device forever, or
    > until the program ends....which ever comes first.
    >
    > This currently works in a non-threaded version, but only for one device
    > at a time, there is a need to create a single windows(yeach) service
    > that talks to many of these devices at once. I don't need worker
    > threads that handle seperate portions of the entire job, i need a
    > single application to spawn multiple processes to run through the
    > entire communication from configure to report, sleep until the next
    > interval time and run again. The communication could last from 1
    > minute to 10 minutes before it ends.


    8<------------------------------------------------------------------------------

    not sure if I understand this correctly - but I would in this position spawn new
    threads, and use a global list of queues to interface between the new threads
    and the old comms module, still talking to one device at a time, but now time
    sliced. - in the comms module:

    for q in list_of_queues:
    see if anything to ask:
    continue if not
    ask it and put answer on reply queue

    but then I am a Philistine coder, interested only in getting the job done...

    - Hendrik
     
    Hendrik van Rooyen, Oct 20, 2006
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.

Share This Page