Storing data periodically on remote server

[carmelo]

Hi everybody,
I have to develop a java application to send every 10 seconds a tuple,
made of 3-4 fields, to a remote server. I thought to proceed in this
way:
- server side: mysql dbms (for storing clients transmitted data)
- client side: java application which makes an odbc connection to the
remote mysql db

The expected workload is quite high, in fact, there will be about 100
clients, each of them send data on time interval of 10 seconds.

Do you think is it a good solution?

Thank you very much
Carmelo

 

[Peter D.]

Hi everybody,
I have to develop a java application to send every 10 seconds a tuple,
made of 3-4 fields, to a remote server. I thought to proceed in this
way:
- server side: mysql dbms (for storing clients transmitted data)
- client side: java application which makes an odbc connection to the
remote mysql db

The expected workload is quite high, in fact, there will be about 100
clients, each of them send data on time interval of 10 seconds.

Do you think is it a good solution?

Thank you very much
Carmelo

I don't know if connecting to the DB directly is a good idea. I would
recommend creating a web service to accept the data instead. The
client will connect to the web service and send the info. The server
side web service will then connect to the DB and do what it needs to
do.

With a web service you can even give your clients the option of coding
their own solutions with any language. Also, it gives you free reign
on your DB server since the web service will abstract it from the
users. If you want to change DBMS you just need to modify the web
service and not every single client.

I think providing a standard web service interface is the way to go.

 

[Arne Vajhøj]

I have to develop a java application to send every 10 seconds a tuple,
made of 3-4 fields, to a remote server. I thought to proceed in this
way:
- server side: mysql dbms (for storing clients transmitted data)
- client side: java application which makes an odbc connection to the
remote mysql db

The expected workload is quite high, in fact, there will be about 100
clients, each of them send data on time interval of 10 seconds.

Do you think is it a good solution?

I assume you mean JDBC connection and not ODBC connection.

100 INSERT per 10 seconds = 10 INSERT per second

is really a very low volume for a real database. You MySQL
server should not have any problems with that.

I assume that open ports and database security are covered.

Arne

 

[carmelo]

I assume you mean JDBC connection and not ODBC connection.

100 INSERT per 10 seconds = 10 INSERT per second

is really a very low volume for a real database. You MySQL
server should not have any problems with that.

I assume that open ports and database security are covered.

Arne

Yes, I mean a JDBC:ODBC connection using the mysql connector jar

 

[carmelo]

I don't know if connecting to the DB directly is a good idea. I would
recommend creating a web service to accept the data instead. The
client will connect to the web service and send the info. The server
side web service will then connect to the DB and do what it needs to
do.

With a web service you can even give your clients the option of coding
their own solutions with any language. Also, it gives you free reign
on your DB server since the web service will abstract it from the
users. If you want to change DBMS you just need to modify the web
service and not every single client.

I think providing a standard web service interface is the way to go.

Thank you for your interesting reply.
But, do you think that using web services will cause problems having
100 clients which connects to the server every 10 seconds? Because
using web services it will be more flexible, but I'll have another
layer...

What do you think about?

 

[Peter D.]

Thank you for your interesting reply.
But, do you think that using web services will cause problems having
100 clients which connects to the server every 10 seconds? Because
using web services it will be more flexible, but I'll have another
layer...

What do you think about?

I understand your concerns and Lew is correct but your load is
actually quite light. 10 connections per second is not very much these
days. Also creating a web service is more work but it is a better
solution all around. It really does depend on your needs though, if
this is just a simple application that you don't see changing much
then by all means connect directly to the DB.

But I really really don't like connecting remote clients (whom I have
no real control over other than the client code) directly to a DB. Not
only do you need to give them all unique accounts you are also
distributing those credentials to each client. If that information
gets out, you could have people trying to access your database
directly.

In the long run, if this service is going to grow or be used for
highly sensitive or critical information a web service is the way to
go.

 

[John B. Matthews]

Even if you didn't use web services you'd still be able to keep up
with such a light load.


Web services increase scalability.

OP: This is crucial. If your 100-client, 0.1Hz solution is good, someone
may want 1000 clients and 1Hz data. [I reiterate the point only because
I may have needed some convincing in the remote past.]

Also, consider buffering data in the client, as a function of service
level and criticality.

 

[Lord Zoltar]

I assume you mean JDBC connection and not ODBC connection.

100 INSERT per 10 seconds = 10 INSERT per second

is really a very low volume for a real database. You MySQL
server should not have any problems with that.

I assume that open ports and database security are covered.

Arne

Of course, if the stress-testing resources are available, I'd
recommend cranking it up 100's or 1000's per second, to figure out
just how far the system can go before it breaks. Could help plan for
future expansion and you'd know the maximum number of clients (or
messages) you can handle.

 

[Arne Vajhøj]

Yes, I mean a JDBC:ODBC connection using the mysql connector jar

The MySQL connector jar is pure JDBC - no ODBC involved.

Does your connection URL start with jdbcdbc: or with jdbc:mysql: ?

Arne

 

[Arne Vajhøj]

I don't know if connecting to the DB directly is a good idea. I would
recommend creating a web service to accept the data instead. The
client will connect to the web service and send the info. The server
side web service will then connect to the DB and do what it needs to
do.

With a web service you can even give your clients the option of coding
their own solutions with any language. Also, it gives you free reign
on your DB server since the web service will abstract it from the
users. If you want to change DBMS you just need to modify the web
service and not every single client.

The extra tier makes it easier to switch database and the use of
SOAP/HTTP as transport makes it possible to write clients in
other languages.

But are those features needed ?

If they are, then it is a good way.

But if they are not then it is just adding lots of
unnecessary complexity to the solution.

Arne

 

[Arne Vajhøj]

I started reading this dubiously, but came around as I pondered.

There definitely is a learning curve, although Axis at least tries to
make the entry barrier quite low. The benefit of thinking of your
system as a group of services is incalculable. Never mind the "web"
part, the cooperating-service model is good to know.

While 100 clients pulsing once every ten seconds is not actually a very
demanding scenario, it still benefits from a web-service architecture,
properly constructed. I don't even think it's much harder to conceive
than a more monolithic local app would be.
>
For example, Peter's suggested data-delivery service would be quite
monomaniacal. It would understand the data needs, in terms of the
problem domain of course, but only as relevant data. It wouldn't care
what the client code logic does with that data. Just concentrate on how
to get to the data store and construct the desired response. The
clients would remain ignorant of the dirty details - as Peter says, they
have been abstracted.

Application development shifts to a mentality of pulling needed action
or information from services that are lying around awaiting activation.

Web services are a relatively easy way to get into service architecture,
not only as a programmer but in that you can deploy on as little as Tomcat.

Deployment turns out to be the bitch.

Exposing data access layer as services is a well known anti-pattern
in SOA.

You should expose the business logic as services.

There may not be much business logic in this case, so it is more
a matter of English than of Java.

Arne

 

[Arne Vajhøj]

Thank you for your interesting reply.
But, do you think that using web services will cause problems having
100 clients which connects to the server every 10 seconds? Because
using web services it will be more flexible, but I'll have another
layer...

What do you think about?

10 web service calls per second is not a big load either.

But it will use much more CPU than the JDBC call (think x10-25).

Arne

 

[Arne Vajhøj]

Even if you didn't use web services you'd still be able to keep up with
such a light load.

Even if using web services he will be able to keep up with that load.

Web services is close to the most expensive transport in existance
(technology independence comes at a cost).

Web services increase scalability.

No - not really.

Scalability depends on the whether every tier behind
is horizontally scalable and stateless.

Changing the transport to SOAP/HTTP or introducing a
new tier with that transport does not change anything.

Arne

 

[Arne Vajhøj]

Even if you didn't use web services you'd still be able to keep up
with such a light load.

Web services increase scalability.

OP: This is crucial. If your 100-client, 0.1Hz solution is good, someone
may want 1000 clients and 1Hz data. [I reiterate the point only because
I may have needed some convincing in the remote past.]

Yes. But it does not help at all.

If the database server can handle 1000 INSERT per second, then
putting 4 app servers in front of it that can each handle 250
web service calls per second is not increasing scalability at all.

Arne

 

[John B. Matthews]

carmelo wrote:
But, do you think that using web services will cause problems
having 100 clients which connects to the server every 10 seconds?
Even if you didn't use web services you'd still be able to keep up
with such a light load.

Because using web services it will be more flexible, but I'll have
another layer...
Web services increase scalability.

OP: This is crucial. If your 100-client, 0.1Hz solution is good, someone
may want 1000 clients and 1Hz data. [I reiterate the point only because
I may have needed some convincing in the remote past.]

Yes. But it does not help at all.

If the database server can handle 1000 INSERT per second, then
putting 4 app servers in front of it that can each handle 250
web service calls per second is not increasing scalability at all.

I appreciate your insights on this; can you elaborate? You're saying
it's not the web service approach itself that increases scalability?
Servlets would perhaps do as well with less overhead?

 

[John B. Matthews]

I appreciate your insights on this; can you elaborate? You're
saying it's not the web service approach itself that increases
scalability? Servlets would perhaps do as well with less overhead?

Web services are often implemented via servlets. These are not
distinct solutions.

There's scalability and there's scalability. Arne will explain his
take on it, but he's already pointed out the notion that data
services won't work well, or affect scalability at the data layer.

At the business-logic layer, where Arne stated:

You should expose the business logic as services.

t has been my observation that you get certain kinds of
scalability with web services, or more generally, with a SOA. You
won't necessarily get performance scalability but you will get
functional scalability, in that you can add to the uses of your
business components with relatively low effort.

Ah, thank you. This is the distinction I was curious about. In my
experience, collecting large volumes of telemetric data proved laborious
but straightforward. It was the ever-growing desire for queries,
reports, charts and assorted transformations that demanded functional
scalability.

Performance scalability is affected by the separation of components,
a key feature of service-oriented architectures but not limited to
them. If the data layer is properly separated, you can increase
performance by adding more RAM to the database server, speeding up
the CPU, buying faster hard drives, and so forth, without concern for
impact on other subsystems. You can add more application servers,
potentially increasing your ability to handle concurrent clients,
assuming you haven't saturated the data layer or there are processes
that don't make heavy use of it.

Web services provide one way to reduce physical colocation of
resources, and to abstract the dependencies between components.
It's not the only way.

****

I take issue with the notion of a "web[-]service approach" - there's
a service-oriented approach of which a web service system would be a
manifestation. The service approach works across a variety of
implementation strategies.

Point taken.

 

[Arne Vajhøj]

But, do you think that using web services will cause problems
having 100 clients which connects to the server every 10 seconds?
Even if you didn't use web services you'd still be able to keep up
with such a light load.

Because using web services it will be more flexible, but I'll have
another layer...
Web services increase scalability.
OP: This is crucial. If your 100-client, 0.1Hz solution is good, someone
may want 1000 clients and 1Hz data. [I reiterate the point only because
I may have needed some convincing in the remote past.]

Yes. But it does not help at all.

If the database server can handle 1000 INSERT per second, then
putting 4 app servers in front of it that can each handle 250
web service calls per second is not increasing scalability at all.

I appreciate your insights on this; can you elaborate? You're saying
it's not the web service approach itself that increases scalability?
Servlets would perhaps do as well with less overhead?

It should be obvious from my example above that the 4 app servers
in front of the database server does nothing for scalability of
the entire solution, because the database server is not scaled
with the web service.

The server side of a web service will if it is implemented in Java
be a servlet.

A servlet that took a POST with a BLOB in request body and returned
a BLOB in response body would be more efficient than a web service
call. But also more difficult to code against in and could even be
impossible to code against from other languages than Java if it
contained Java specific stuff.

But neither improve scalability of the solution.

Arne

 

[Arne Vajhøj]

John B. Matthews wrote:
At the business-logic layer, where Arne stated:

it has been my observation that you get certain kinds of scalability
with web services, or more generally, with a SOA. You won't necessarily
get performance scalability but you will get functional scalability, in
that you can add to the uses of your business components with relatively
low effort.

Performance scalability is affected by the separation of components, a
key feature of service-oriented architectures but not limited to them.
If the data layer is properly separated, you can increase performance by
adding more RAM to the database server, speeding up the CPU, buying
faster hard drives, and so forth, without concern for impact on other
subsystems. You can add more application servers, potentially
increasing your ability to handle concurrent clients, assuming you
haven't saturated the data layer or there are processes that don't make
heavy use of it.

Web services provide one way to reduce physical colocation of resources,
and to abstract the dependencies between components. It's not the only
way.

The term scalability is usual used in relation to performance not
functionality.

Stateless app servers scale horizonal while database servers (excluding
really highend solutions like Oracle RAC) scale vertical.

Arne

 

[John B. Matthews]

John B. Matthews wrote: [...]

I appreciate your insights on this; can you elaborate? You're
saying it's not the web service approach itself that increases
scalability? Servlets would perhaps do as well with less overhead?

It should be obvious from my example above that the 4 app servers
in front of the database server does nothing for scalability of
the entire solution, because the database server is not scaled
with the web service.

The server side of a web service will if it is implemented in Java
be a servlet.

A servlet that took a POST with a BLOB in request body and returned
a BLOB in response body would be more efficient than a web service
call. But also more difficult to code against in and could even be
impossible to code against from other languages than Java if it
contained Java specific stuff.

But neither improve scalability of the solution.

Thanks! I also see now the distinction between vertical and horizontal
scalability that you made up-thread.

 

[Arne Vajhøj]

Thanks! I also see now the distinction between vertical and horizontal
scalability that you made up-thread.

That distinction is critical.

1 box with M CPU -> N boxes with M CPU each cost N times as much money

1 box with M CPU -> 1 box with N*M CPU cost a lot more than N times as
much money

Arne