communication between mini-modules

[Emmanuel Charruau]

Hi,

I am looking for a class or any information which would allow me to
make communicate mini-module in c++. I have been looking

on the net for some examples of such implementation, but I did not
find any good one. This is why I am asking here if some

of you know where to find some information on the internet about such
implementation.

I explain you what I mean per communication between mini-modules.

I want to implement an application for example composed from modules
A,B,C,D and E.

Module A : simply reads the content of a file per n bytes
Module B : convert the binary content in integers numbers text
Module C : Insert html tags in text
Module D : Change text color
Module E : Display text on the screen

What I want to do is using a communication tool (a class) between
these modules, to create dinamicaly a chain which would

allow me to create any combinaision between these modules.

For e.g.

1: A -> B -> C -> D -> E (go back to A)
or
2: A -> B -> D -> C -> E (go back to A)

You can see that the program would be able to first insert the html
tags before to change the color (1), or do it the other

way around (2).

The really interesting part is that the order could be then
configurable and changed whenever a user need to insert a new

module without compliling a new program.

I have been searching extensively the internet, believing several
times that I found the solution.
I though CORBA was the solution, but it seems that even if it could do
the job, the kde team as abandonned it because it was

too slow.
I tryed communicating through the pipe and using stdin stdout, but
even if this works, that is a one way channel and does

not allow complex connection between modules.
I have tryed using soap but this is definitly to slow.
I have tryed using ip (using sockets) but this is not the best
solution if all the modules are on the same machine (though

this is quite good if the modules are on separate machines).

Thanks in advance,

Emmanuel Charruau

 

[shp]

Hi!

I am in a similar situation, wanting to dynamically be able to chain up
different modules and have them communicate.

I have considered using the signals/slots paradigm for the
communication. Have you considered that Emmanuel?

Cheers,
Kim

 

[Ivan Vecerina]

I am looking for a class or any information which would allow me to
make communicate mini-module in c++. I have been looking
on the net for some examples of such implementation, but I did not
find any good one. This is why I am asking here if some
of you know where to find some information on the internet about
such implementation.

I explain you what I mean per communication between mini-modules.

[...]
Which looks pretty much like standard unix pipes: you create
multiple separate programs that read from their standard input
and write to their standard output, and then a shell can be used
to 'pipe' data through a set of programs in any order you want.

The really interesting part is that the order could be then
configurable and changed whenever a user need to insert a new
module without compliling a new program.

I have been searching extensively the internet, believing several
times that I found the solution.
I though CORBA was the solution, but it seems that even if it could
do the job, the kde team as abandonned it because it was too slow.
I tryed communicating through the pipe and using stdin stdout, but
even if this works, that is a one way channel and does
not allow complex connection between modules.

What type of connections are you looking for? Why is a one-way
channel not good enough?

I have tryed using soap but this is definitly to slow.
I have tryed using ip (using sockets) but this is not the best
solution if all the modules are on the same machine (though
this is quite good if the modules are on separate machines).

The key questions are:
- what are the kinds of messages you want to send between the
objects, and in what directions?
- how do you want to schedule the processing of the data?

For the latter, do you want a chunk of data provided as input to
be immediately processed all the way down the chain, or are you
looking for a somewhat more asynchronous processing?

I mean, in the simplest case you could just use a base class such as:

// base class for an object that receives data
class DataHandler
{
public:
virtual void process(std::string const& data)=0;
virtual ~DataHandler() {}
}

// base class for a data processing filter
class DataPipe : public DataHandler
{
public:
DataPipe(DataHandler& next) : next_(next) {}
virtual void process(std::string const& data)=0;
//NB: implementations shall call forward() to pass data further
protected:
void forward(std::string const& data) { next->process(data); }
private:
DataHandler& next_;
};

You may also replace the 'push' model above with a 'pull'
model, where the end-user of the data requests for the
next 'packet' from a (chained) data source.
Another approach would be to 'post' data packets to be processed
to a global queue of <data,object&> pairs, and use some kind of
scheduling (possibly distributing the processing to multiple
threads).

There are many related techniques, design patterns, and libraries
that can be used. Similar architectures are used for implementing
drivers, for multimedia frameworks (e.g. QuickTime), and many
other problem domains. You haven't provided a sufficient level
of detail to allow the selection of a specific approach.


Cheers,
Ivan

 

[Emmanuel Charruau]

Hi Thanks a lot fot your answer,
I answer your question in the text

Which looks pretty much like standard unix pipes: you create
multiple separate programs that read from their standard input
and write to their standard output, and then a shell can be used
to 'pipe' data through a set of programs in any order you want.

Exactly and this is one of the goal of the architecture. One of the
goal is like I wrote before is to be able to interconnect different
modules together in different orders, but an other one is to try to
keep the unix way of working, small parts but which works fine. This
way if we create a new module we can concentrate our testing mainly on
this module and not on th whole application.

The key questions are:
- what are the kinds of messages you want to send between the
objects, and in what directions?
- how do you want to schedule the processing of the data?

This will mainly be big bunch of data (1000 lines of text data in a
row) not so often, or very small amount of data (1 line of text to
filter) very often, this depending on the way I will find to
implement.
The application must analyse text lines with number in it (could be a
list of telephone call for example). And apply rules on the lines it
decided to filter (for example all the telephone call starting with
12345). The rule can be for example to add 0033 to the beginning of
the filtered number to tell it is a french number.

I could implement it easily in a monolityque way, but this would
really be interesting to do it per separate programs, to be able to be
very flexible to assemble the programs in different order.

I give you an example where it would be extremly useful. I have a
module A which read a text file line by line, A->B. The module B
filters all the numbers starting with 12345 then gives the lines
containing these numbers to C. C look if the number is a french
number. If it is a french number the text line is given to D and a
copy of the line is slighly modified and sent again to B, being
filtered again.


About the scheduling, I was thinking that a if we process the data
line by line each line would go at a time through the modules, and
wait that the next module is finished with it works to be pushed.
Maybe testing a "I am ready to process" flag.

For the latter, do you want a chunk of data provided as input to
be immediately processed all the way down the chain, or are you
looking for a somewhat more asynchronous processing?

I mean, in the simplest case you could just use a base class such as:

// base class for an object that receives data
class DataHandler
{
public:
virtual void process(std::string const& data)=0;
virtual ~DataHandler() {}
}

Here I think I understand, for every new module I can create a new
class derived from DataHandler and implement a specific virtual
function corresponding to this process. Which allows me to "plug" new
modules (even without recompiling the project if I recall the theory).

// base class for a data processing filter
class DataPipe : public DataHandler
{
public:
DataPipe(DataHandler& next) : next_(next) {}
virtual void process(std::string const& data)=0;
//NB: implementations shall call forward() to pass data further
protected:
void forward(std::string const& data) { next->process(data); }
private:
DataHandler& next_;
};

Ok it took me more time to understand this one.
I derive a class DataPipe from the base class DataHandler.
Then I initialise the variable next_ with the value given in
DataPipe(next). _next being in fact the next module (process) or its
pointer at least.
Then Forward executes the next function.
I have to try that definitly!

You may also replace the 'push' model above with a 'pull'
model, where the end-user of the data requests for the
next 'packet' from a (chained) data source.
Another approach would be to 'post' data packets to be processed
to a global queue of <data,object&> pairs, and use some kind of
scheduling (possibly distributing the processing to multiple
threads).

There are many related techniques, design patterns, and libraries
that can be used. Similar architectures are used for implementing
drivers, for multimedia frameworks (e.g. QuickTime), and many
other problem domains. You haven't provided a sufficient level
of detail to allow the selection of a specific approach.

Are the info I gave you enough to choose a specific design patterns,
and library?

Really thanks a lot for the answer you already gave me answer, it
already proves me that the realisation is possible. It is even better
to know that some technics are available somewhere.

Regards,

Emmanuel

 

[Ivan Vecerina]

This will mainly be big bunch of data (1000 lines of text data in a
row) not so often, or very small amount of data (1 line of text to
filter) very often, this depending on the way I will find to
implement.
The application must analyse text lines with number in it (could be a
list of telephone call for example). And apply rules on the lines it
decided to filter (for example all the telephone call starting with
12345). The rule can be for example to add 0033 to the beginning of
the filtered number to tell it is a french number. ....
About the scheduling, I was thinking that a if we process the data
line by line each line would go at a time through the modules, and
wait that the next module is finished with it works to be pushed.
Maybe testing a "I am ready to process" flag. ....
Are the info I gave you enough to choose a specific design patterns,
and library?

I think so - the lines can be processed independently, so this
should be a relatively simple case.
The example I provided should work. One potential inconvenience
of it is that the message is forwarded recursively (shouldn't matter,
really, can even be good for debugging/flexibility, but some would
argue that this is suboptimal for long chains of filters).

An alternative approach would be something like:
// base class for all data 'filters'
class Filter {
public:
// modifies line, returns false to cancel further processing.
virtual bool transform(std::string& line)=0;
virtual ~Filter() {}
};
class MultiFilter : public Filter
{
public:
MultiFilter() {}

// to add a 'filter' to the chain
void appendFilter( Filter& filter ) { filters_.push_back(&filter); }

// implementation of Filter interface:
bool tranform(std::string& line)
{
for( Filters::iterator scan = filters_.begin()
; scan != filters_.end() ; ++filters_ )
{
if( ! (**scan).transform(line) )
return false;
}
return true;
}
private:
typedef std::vector<Filter*> Filters;
Filters filters_;
}

void applyTransform( std::istream& source
, std:stream& destination
, Filter& filter )
{
std::string line;
while( std::getline(source,line) )
{
if( filter.tranform(line) )
destination << line << '\n';
}
}

And the top-level code could be something like:
int main()
{
MultiFilter mf;
FilterA fa; // a subclass of Filter
mf.appendFilter( fa );
FilterB fb; // etc...
mf.appendFilter( fb );
applyTransform(std::cin, std::cout, mf);
}

Really thanks a lot for the answer you already gave me answer, it
already proves me that the realisation is possible. It is even better
to know that some technics are available somewhere.

From your description, it still looks like you could use sed or
a Perl script but the base infrastructure is easy to implement
in C++.


Regards,
Ivan

 

[Emmanuel Charruau]

Hi Ivan,

Thanks again for your answer. I will try to implement this solution
next week and I will tell you the result.

Emmanuel

 

[Emmanuel Charruau]

Hi Kim,

I will first try Ivan solution, but this seems a good idea, and
knowing that I am using qt to program that would be an easy solution
to try.
Thanks for the hint.

Regards,

Emmanuel

 

[Emmanuel Charruau]

Hi Ivan,

I implemented your solution and it works perfectly. I slightly
modifyed the project by replacing the loop scanning the vector by a
mapping letting the user choosing the filter execution the order he
needs.

What I will try now to do is to find how to make the differents
modules working at the same time. Let say if module A has already
given its informations to module B it can process some new information
at the same time module B is processing its data.

Regards,

Emmanuel

 

[Emmanuel Charruau]

Hi,

I have been looking the web to find a solution to run multiple modules
(process) at the same time. I found out that I could use
multithreading, but I also have to manage scheduling the messages
communication between the modules.
What I would like to know is, does it exist a C++ library helping to
manage running several processes at the same time and scheduling some
messages communication between the modules. All that in the same C++
program? If not, can somebody suggests some books explaining how to
create a program allowing this?

Regards,

Emmanuel

 

[Ivan Vecerina]

(re)bonjour Emmanuel,

I have been looking the web to find a solution to run multiple modules
(process) at the same time. I found out that I could use
multithreading, but I also have to manage scheduling the messages
communication between the modules.
What I would like to know is, does it exist a C++ library helping to
manage running several processes at the same time and scheduling some
messages communication between the modules. All that in the same C++
program? If not, can somebody suggests some books explaining how to
create a program allowing this?

First of all, you need to consider the fact that unless the
program is running on a mutliprocessor/multicore or at least
hyperthreaded machine, it is unlikely to run faster (just)
because you use multithreading or multiprocessing.

Second, multi-threaded programming is notoriously difficult.
Especially in performance-critical applications!

The proper way to share the load between multiple threads
depends a lot on the specific of the application, i.e. the
kind of data source and data destination being used.
Ways to parallelize the processing of your chunks of data
you use include:
- Splitting the chain of filters in two, with a single
(serialized) queue of data chunks used for data exchange.
This would be related to a producer/consumer pattern.
What is difficult here is to find the right split point
so that both threads work at full capacity.
- Having two identical threads being feeding themselves
from a common data source, and sending results to the
same output. What has to be watched for in this case
is that all accesses to the source/destination need
to be serialized, and some extra logic may be required
if the data chunks must be written to the destination
in the same order as they were written in the source.
In either of the above cases, you could use more that
two threads if this is supported by your system.

Multithreaded programming in C++ requires you to rely
on platform-specific libraries, although some cross-platform
wrappers exist as well (such as boost::threads, see
http://www.boost.org/doc/html/threads.html).
Some pointers to references are provided at the bottom
of that page. I liked Schmidt's "Pattern-Oriented (System)
Architecture" (volume 2).


I hope this helps,
Ivan