Twisted for non-networking applications

K

Kottiyath

Hi all,
Is it a good idea to use Twisted inside my application, even though
it has no networking part in it?
Basically, my application needs lots of parallel processing - but I
am rather averse to using threads - due to myraid issues it can cause.
So, I was hoping to use a reactor pattern to avoid the threads. I am
using twisted in another part of the application for networking, so I
was hoping to use the same for the non-networking part for reusing the
reactor pattern.
If somebody can help me on this, it would be very helpful.
Regards
K
 
J

James Mills

Hi all,
Is it a good idea to use Twisted inside my application, even though
it has no networking part in it?
Basically, my application needs lots of parallel processing - but I
am rather averse to using threads - due to myraid issues it can cause.
So, I was hoping to use a reactor pattern to avoid the threads. I am
using twisted in another part of the application for networking, so I
was hoping to use the same for the non-networking part for reusing the
reactor pattern.
If somebody can help me on this, it would be very helpful.

Alternatively you could give circuits (1)
a go. It _can_ be a nice alternative to
Twisted and isn't necessarily focused on
Networking applications.

cheers
James

1. http://trac.softcircuit.com.au/circuits/
 
R

RajNewbie

Alternatively you could give circuits (1)
a go. It _can_ be a nice alternative to
Twisted and isn't necessarily focused on
Networking applications.

cheers
James

1.http://trac.softcircuit.com.au/circuits/

I was unable to see documentation explaining this - so asking again.
Suppose the event handlers in the component is doing blocking work,
how is it handled?
I went through ciruits.core, but was unable to understand exactly how
blocking mechanisms are handled.
My scenario is as follows:
I have 4 loops, 1 small and high priority, 3 quite large and blocking
(takes upto 3 seconds) and comparatively low priority.
The small loops goes through everytime and does some work - and
optionally uses the data sent by the other 3 loops.
I do not want the smaller loop to get blocked by the other loops.

So, if the event handler does blocking work, can that cause the whole
loop to block?
 
J

James Mills

I was unable to see documentation explaining this - so asking again.

Documentation is available here:
http://trac.softcircuit.com.au/circuits/wiki/docs
And here: pydoc circuits

The code itself is heavily documented. I'm still
writing better online references, tutorials
and what not ... :) Please see the examples/
Suppose the event handlers in the component is doing blocking work,
how is it handled?

You have a few options.
1. Do your work in a thread.
2. Do your work in a process.

You could utilize the Worker component for
such things. I'm also building a Process
component that uses the multiprocessing module.
I went through ciruits.core, but was unable to understand exactly how
blocking mechanisms are handled.

They aren't. It's up to you to handle such
situations. If your "event-handler" blocks,
everything, it will block every other event
handler from being processes.

In what situation would you have this ?
I'm curious :)
My scenario is as follows:
I have 4 loops, 1 small and high priority, 3 quite large and blocking
(takes upto 3 seconds) and comparatively low priority.
The small loops goes through everytime and does some work - and
optionally uses the data sent by the other 3 loops.
I do not want the smaller loop to get blocked by the other loops.

This sounds complex :) What is your application ?
What's being processes ?
So, if the event handler does blocking work, can that cause the whole
loop to block?

Yes. If you decide to use circuits, I suggest
you restructure your program. Try to do
your work (if it's blocking) in Worker components (threads).

cheers
James
 
B

Bryan Olson

Kottiyath said:
Is it a good idea to use Twisted inside my application, even though
it has no networking part in it?
Basically, my application needs lots of parallel processing - but I
am rather averse to using threads -

With or without threads, the Python interpreter does not do parallel
processing. You could use multiple processes, or a thread could call an
extension module that releases Python's global interpreter lock, but
Python itself does not offer parallel processing.
due to myraid issues it can cause.
So, I was hoping to use a reactor pattern to avoid the threads.

The reactor pattern describes event-driven I/0, not parallel processing.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

A Twisted Design Decision 0
Twisted 13.2.0 Release Announcement 0
twisted server 0
Twisted: 1 thread in the reactor pattern 5
ANN: Twisted 9.0.0 0
HTTPS on Twisted 2
Twisted on Windows 1
Networking! 1

Members online

Forum statistics

Threads
473,733
Messages
2,569,440
Members
44,831
Latest member
HealthSmartketoReviews

Latest Threads

Top