Archiarchitecture Network, need some help!

[Diego Bernardes]

This is my problem.

I have a universe of 10k gprs modems connecting on my server, they
connect on the same port and ip. After the connection, the modem send a
identification string, with this string i can authenticate the modem and
create a internal tcp port, with this port my internal application can
send data to the modem and the modem to the application. This is the
architecture that we have today but is using a closed source software
and its giving us alot of trouble.

So im thinking in do this with EventMachine:

One eventmachine server listening on port 10000 (where the gprs modems
will connect).
x eventmachine servers based on the gprs modems registred on database.

What i need is to communicate between the port 10000 and the internal
port, need to be secure connection, i cant lose the data, maybe
RabbitMQ, but everytime i try to use it i get errors .... >.<
And i dont want to create another eventmachine server to communicate
with port 10000 and to comunicate with the internal port, more 2 tcp
servers per connection.

I need get 10k connections per server and after this i gonna use a load
balancer

Bellow is a picture of the architecture:
http://www.uploadimagens.com/upload/6dc073f681d0b6ef2210406e8016e6ea.jpeg

Anyone sugest anything better?
Help?

 

[Brian Candler]

Anyone sugest anything better?

The overall approach sounds fine, but from your description it sounds
like you're not exactly clear about what is talking to what. After the
gprs modem connects and authenticates, does is expect to have the same
TCP connection proxied through to another server? Does it submit framed
messages over this TCP connection which you wish to put into a queue? Do
you need to send any messages back to the modem, and if so, where do
they come from? You need to be clear on the difference between
synchronous protocols (strict request/response) and asynchronous.

Perhaps you should start simple. Build something with Gserver which
accepts connection from a handful of modems and processes the messages,
proving you have the right protocol information and logic. Then port it
to Eventmachine if necessary to scale. But it might not be necessary;
you could instead have 20 Ruby processes and have a simple TCP
load-balancer in front like pen, so each process is only handling 500
connections. http://siag.nu/pen/

As for RabbitMQ: if you can't get Ruby to talk to it, then that's a
separate problem which you can solve separately (perhaps ask on a
RabbitMQ list)

There are many other solutions for "can't lose data", for some
definition of "can't lose". e.g. SQL database, couchdb, ...etc

 

[Diego Bernardes]

Let me try to explain better, my english is very bad >.>

Im working with telemetry in a energy company, so we use gprs modems
connected to digital metters to know the consumption of energy, billing,
turn off and on energy in places and so...

We have two softwares to do this, the "X" take care of the comunication
(the one im making) and the "Y" send and take the right comands to do
the job (this server use the one im making to send and get the data).

All the modems connect to a ip and port, inside the server every modem
should have a ip and port so the second software can operate with him.

This can be configurable sending SMS or if the modem is connecting using
GPRS (im gonna use this to do the load balancing making some part of the
modems connect on another server if the primary is overloaded)

When the software i make start, i create the port 10000 and get all the
ports of all modems on database and create them too, so everyport is
listening to new connections.

The port 10000 get the connections of the modems and the other ports get
the connections of the "Y" software.

What i need is, when the server "Y" connect to a specific port and send
a command, this command should be delivered to the rigth modem trough
the port 10000, and when i get the result from the modem send it back to
the application.

The comunication always start from the application, never from the
modem. (at least till now... hehe)

The modems are configurable to stay connected forever, if the connection
drops, it try to connect again.


I think now its clear

 

[Diego Bernardes]

When i said i can't lose the data is, if the application drops the
information i sended is lost, and im not sure if i can recover sending
again the same command.

I need store it in somewhere, database, memcache, message queue,....

Maybe rabbitmq can do what i need.

I can create two channels for each port, ex "50000_in" and "50000_out"
and subscrive the "50000_in" on port 10000, and in the internal port
"50000_out", so everytime i put something from the port 10000 on
"50000_in" the port "50000_out" get the data, and vice versa.

But this is another problem, i need know how many channels can be
created with rabbitmq, because today i have 15k modems so 30k channels
and this number is increasing...