MapReduce?

[jlc488]

I have multiple log files which are between 10Mb to 30Mb.

I need to extract data according to my needs.

For example, I only need to required to have the result of memory
usage or something else.

Someone told me about the Hadoop Map/Reduce. Just wondering, Is It the
right solution to use according to this kind of situation?

Any other suggestions would be really appreciated!

Thanks.

 

[Daniel Pitts]

I have multiple log files which are between 10Mb to 30Mb.

I need to extract data according to my needs.

For example, I only need to required to have the result of memory
usage or something else.

Someone told me about the Hadoop Map/Reduce. Just wondering, Is It the
right solution to use according to this kind of situation?

Any other suggestions would be really appreciated!

Thanks.

10-30mb doesn't warrent that. If it was 10-30gb then maybe. 10-30tb,
then definitely.

 

[Arne Vajhøj]

I have multiple log files which are between 10Mb to 30Mb.

I need to extract data according to my needs.

For example, I only need to required to have the result of memory
usage or something else.

Someone told me about the Hadoop Map/Reduce. Just wondering, Is It the
right solution to use according to this kind of situation?

Any other suggestions would be really appreciated!

With such tiny data you can just store the data in memory on
a single system.

So you need to look at Map/HashMap/List/ArrayList not at
Hadoop.

Arne

 

[Roedy Green]

Someone told me about the Hadoop Map/Reduce. Just wondering, Is It the
right solution to use according to this kind of situation?

Usually these sorts of tools have a free trial.

Then you have to decide if the tool is sufficiently powerful to do
what you need. If it looks as if it will run out of steam, you might
as well right custom code from the start.

Conversely, if it looks vastly overpowered and expensive, chances are
you can whip up something custom more cheaply.

One way to get a lot of fancy searching/consolidating power is to
write a module to export your logs into SQL. This also offload the
problem of clever efficient algorithms on the SQL manufacturer.

There are also general report generating tools that will work with
SQL.
--
Roedy Green Canadian Mind Products
http://mindprod.com

"Let us pray it is not so, or if it is, that it will not become widely known."
~ Wife of the Bishop of Exeter on hearing of Darwin's theory of the common descent of humans and apes.

 

[Arne Vajhøj]

Usually these sorts of tools have a free trial.

Hadoop is open source.

Then you have to decide if the tool is sufficiently powerful to do
what you need. If it looks as if it will run out of steam, you might
as well right custom code from the start.

Conversely, if it looks vastly overpowered and expensive, chances are
you can whip up something custom more cheaply.

It is free.

One way to get a lot of fancy searching/consolidating power is to
write a module to export your logs into SQL. This also offload the
problem of clever efficient algorithms on the SQL manufacturer.

For a problem that really needs the map reduce approach, then
a relational database and SQL is hopeless.

Arne