Selecting Technology for a Data Filtering and Manipulation Project

S

sengsational

New Project: SAP to i2 using Oracle (< sorry to do that to you in a
Java forum).

Last week I played with Oracle's JDeveloper a little bit. This week
I've been playing with Eclipse DTP with Oracle's plug in. I'm sure
I've only scratched the surface of possibilities, so thought I'd see
if there were any ideas from the community (that's you!).

The project is to manage and manipulate data between SAP's Sales and
Distribution (for demand data), various i2 supply chain apps, and back
to SAP Product Planning. Master data and transactional data. For
those coders not into manufacturing, all of that doesn't really
matter. The point is that between these commercial applications will
be periodic batch jobs that involve large amounts of data which needs
heavy manipulation sometimes, needs a significant amount of
persistence (Oracle will be used), and will need many batch jobs to
run in a coordinated way.

One solution would be to write a bunch of PL/SQL procedures and drive
those from an external batch scheduling system. But of course that
produces spaghetti in the way of cursors, temp tables, sub-sub-sub
selects that are impossible to diagnose, and most of all, doesn't
employ Java!

So the question is: What tool-set should be deployed to handle this
project so that it can be treated like a real application, and not as
a heap of steaming "stuff"? Is there a "clean" way to do this? There
are so many projects running on Eclipse, I don't know if any actually
fit this kind of project. Can an OR mapping idea be employed? BPEL
maybe? Which tools should I look at?

--Dale--
 
A

Abhijat Vatsyayan

sengsational said:
New Project: SAP to i2 using Oracle (< sorry to do that to you in a
Java forum).

Last week I played with Oracle's JDeveloper a little bit. This week
I've been playing with Eclipse DTP with Oracle's plug in. I'm sure
I've only scratched the surface of possibilities, so thought I'd see
if there were any ideas from the community (that's you!).

The project is to manage and manipulate data between SAP's Sales and
Distribution (for demand data), various i2 supply chain apps, and back
to SAP Product Planning. Master data and transactional data. For
those coders not into manufacturing, all of that doesn't really
matter. The point is that between these commercial applications will
be periodic batch jobs that involve large amounts of data which needs
heavy manipulation sometimes, needs a significant amount of
persistence (Oracle will be used), and will need many batch jobs to
run in a coordinated way.

One solution would be to write a bunch of PL/SQL procedures and drive
those from an external batch scheduling system. But of course that
produces spaghetti in the way of cursors, temp tables, sub-sub-sub
selects that are impossible to diagnose, and most of all, doesn't
employ Java!

So the question is: What tool-set should be deployed to handle this
project so that it can be treated like a real application, and not as
a heap of steaming "stuff"? Is there a "clean" way to do this? There
are so many projects running on Eclipse, I don't know if any actually
fit this kind of project. Can an OR mapping idea be employed? BPEL
maybe? Which tools should I look at?

--Dale--
Sounds like an data integration/ETL (extraction, transformation and
loading) job for which something like Informatica or its competitors
should be a good choice. There are open source java based tools which I
haven't used but look into clover ETL and Pentaho data integration. And
search for (open source?) data integration tools.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,769
Messages
2,569,581
Members
45,056
Latest member
GlycogenSupporthealth

Latest Threads

Top