Is perl better? :(((

D

dar7yl

Looking at your code, I can see immediately where your problem is.
Remember, perl syntax and java syntax are different.
for (int i=0; i<2000; i++)
ps.setInt(1, i);

this loops around 2000 times, setting the same object. time waster supreme.
Perhaps you meant to put braces {} around a larger section.
conection.getPreparedStatement("select 'key', sysdate+? from dual");

I'm not sure that this is the correct form for what you are trying to do.
You cannot pass the name of a column in a select clause as a parameter
to a prepared statement.

Perhaps you meant to say "select 'key', 'sysdate' from dual."
then execute your query, returning a resultset rs, which you iterate so:

ResultSet rs = ps.executeQuery();
while ( rs.next() )
{
String key=rs.getString(1 );
double value=0;
Double d=(Double)ht.get(key);
if (d!=null) value=d.doubleValue();
value+=rs.getDouble(2);
ht.put(key,new Double(value));
}

regards,
Dar7yl.
 
T

Thomas Kellerer

Luke Webber wrote on 01.02.2005 22:49:
I have one off-the-wall suggestion. Why not try removing the
"rs.close()" statement and see how it goes? I have a niggling feeling
that it might be making an extra round trip to the database server for
some reason.

I don't think that's a good idea. That will leave him with a lot of
unclosed cursors...

Thomas
 
D

dar7yl

Alex:

I think I can see where a lot of your time is being spent - in the sql query
and the subsequent returning of the resultset.

If, as you say, the query is generated the same in java as it is in perl,
then the only differences would be the transmission of the resultset,
as the sql server would be performing the same operation
(nearly enough) in both cases.

If your jdbc driver supports it, turn on tracing/profiling. The MySQL
driver has this feature which you turn on by adding "?&profileSql=true"
to the connect string. This is a great feature, which shows the actual
expanded query, the server execution time, and the transmission time.

I assume that the query returns only one result. It would be better
programming practice to surround the logic with
if (rs.next())
{
// logic goes here
}
instead of performing the continue on the negative case. (bad style).
Even better would be to make that a while, just in case the query
returns more than one result.

Perhaps, you could optimize the query to return the whole list in one call.
A way to do that would be to create a temporary table from your getData()
table and then do a join in your sql query, thus returning a dataset with
more than
one row. This would take a great deal of load off the server, as it would
only
have to perform the scan once (albeit a more complex query).

regards,
Dar7yl

Luke:
we're trying to help
And I really appreciate it.
I'm only a little bit upset when people say that I'm unwiling to
work...

Let's start again with willings.

Profile doesn't have sense for me. It's obvious.
We are waiting for Oracle answer:
java -Xrunhprof:cpu=samples -classpath .:eek:jdbc14.jar DA
rank self accum count trace method
1 74.08% 74.08% 889 102 java.net.SocketInputStream.socketRead0
2 5.42% 79.50% 65 133
java.net.SocketOutputStream.socketWrite0
3 1.33% 80.83% 16 145 oracle.jdbc.ttc7.TTCItem.unmarshal
4 1.17% 82.00% 14 4 java.lang.ClassLoader.defineClass0
5 0.92% 82.92% 11 19
java.lang.ClassLoader.findBootstrapClass
6 0.83% 83.75% 10 143
oracle.jdbc.ttc7.MAREngine.unmarshalCLRforREFS
7 0.58% 84.33% 7 159
oracle.jdbc.ttc7.TTCAdapter.createNonPlsqlTTCColumnArray
8 0.58% 84.92% 7 150
oracle.jdbc.ttc7.TTC7Protocol.createDBItem
9 0.50% 85.42% 6 158 java.lang.StringBuffer.toString

I do not believe in profiling. It takes time.
And during this time Oracle is working too!
So, I prefer more simple way - just count milliseconds.
For whole program, for database, (for sql), for cycle.
See code.

And here is exact code (except sql):
//========= DA.java =============

import java.io.*;
import java.sql.*;
import java.util.*;

public class DA {
public Connection con;
public Statement stmt;

static {
try {
Class.forName("oracle.jdbc.driver.OracleDriver");
} catch (Exception ex) {
System.out.println("static driver not found: "
+ ex.getMessage());
}
}

public void openCon(String url,
String user, String password) {
try {
con = DriverManager.getConnection(url, user, password);
stmt = con.createStatement();
} catch (Exception ex) {
System.out.println(ex);
}
}

public void closeCon() {
try {
stmt.close();
con.close();
} catch (Exception ex) {
System.out.println(ex);
}
}

public static void main(String[] args) {
new DA().getData();
System.out.println("Java="+System.getProperty("java.version"));
}

public PreparedStatement getPreparedStatement(String sql)
throws java.sql.SQLException {
return con.prepareStatement(sql);
}

//////////// method itself ///////////////
public Vector table; // vector of String[4]; 2000 elements
// collecting_code|userid|start|end
public Hashtable totaled_hrs=new Hashtable();
public void getData()
{
try{

double mth_hours=23*7.75; // for December 2004
String oldCode=null;

table=new Vector();
String tmp[]=null;
BufferedReader br=new BufferedReader(new FileReader("load"));
String line;
while ((line=br.readLine())!=null){
StringTokenizer st=new StringTokenizer(line,"|");
tmp=new String[4];
tmp[0]=st.nextToken();
tmp[1]=st.nextToken();
tmp[2]=st.nextToken();
tmp[3]=st.nextToken();
table.add(tmp);
}
br.close();
System.out.println(table.size());
long tall,tdb,tsql,tcycle,v2,v3;
tall=tdb=tsql=tcycle=0;

DA da=new DA();
ResultSet rs;
Properties pr=new Properties();
pr.load(new FileInputStream("common.properties"));

//openCon("jdbc:eek:racle:thin:mad:yahoo.com:1521:users",
// "scott","tiger");

da.openCon(pr.getProperty("url"),
pr.getProperty("user"),pr.getProperty("password"));

double value=0,total=0;
String type="";

String sql=
"SELECT SUM(DECODE(TO_CHAR(e.date, 'MM'), \n"+
" '12', e.hours,0)) hours, \n"+
" d.person_type \n" +
" FROM projects a, tasks b, assignments c, \n"+
" resources d, hours e, \n"+
" super_projects h, users g \n"+
" WHERE \n"+
" d.id = ? \n"+
" AND (e.date >= TO_DATE(?, 'mm/dd/yyyy')) \n"+
" AND (e.date <= TO_DATE(?, 'mm/dd/yyyy')) \n"+
" AND a.id = b.projectid \n"+
" AND b.id = c.taskid \n"+
" AND c.resourceid = d.id \n"+
" AND c.id = e.assignmentid \n"+
" AND e.approved = 'yes' \n"+
" AND a.superid = h.id \n"+
" AND h.name like 'Billed %' \n"+
" AND a.name not like 'Local %' \n"+
" GROUP BY d.person_type \n";

PreparedStatement ps = da.getPreparedStatement(sql);
tall=System.currentTimeMillis();

v2=System.currentTimeMillis();

for(int i=0; i<table.size(); i++)
{
tmp=(String[])(table.get(i));
v3=System.currentTimeMillis();
tcycle+=v3-v2;

ps.setString(1, tmp[1]);
ps.setString(2, tmp[2]);
ps.setString(3, tmp[3]);
rs = ps.executeQuery();
tsql+=System.currentTimeMillis()-v3;
if (!rs.next()) continue;
value=rs.getDouble(1);
type=rs.getString(2);
rs.close();

v2=System.currentTimeMillis();
tdb += v2-v3;

if (value>mth_hours&&"Billed".equals(type)) value=mth_hours;

total+=value;
MyDouble d=(MyDouble)totaled_hrs.get(tmp[0]);
if (d!=null) d.value+=value;
else {
d=new MyDouble();
d.value=value;
totaled_hrs.put(tmp[0],d);
}
if (!(tmp[0].equals(oldCode))&&oldCode!=null)
System.out.println(
oldCode+" "+
((MyDouble)totaled_hrs.get(oldCode)).value+
" "+total);
oldCode=tmp[0];
}// end cycle

da.closeCon();
tall=System.currentTimeMillis()-tall;

System.out.println("Hours: "+total+
" time all="+tall+
" time db="+tdb+
" (sql="+tsql+")"+
" time cycle="+tcycle);
} catch (Exception e){
e.printStackTrace();
}
}

class MyDouble{public double value=0;}

}
//////////// end DA.java /////////////

I hope it will post well. I'll double check it and
if it will be bad I'll put it somewhere in my web site.

And here is perl code:
################# perl ########################
#!/perl/5.6/bin/perl
use lib "/perl/5.6/DBD/oracle8.1.6/lib/site_perl/5.6.0/sun4-solaris";
use Oraperl;

... open connection ....

sub getData{

my(@persons)=@_;
$total=0;
%totaled_hrs=();

foreach $key(@persons){

@one_person=split(/:/,$key);

$sql=...;

$db_list=&ora_open($lda,$sql) || die "\nCan't open cursor
1: $ora_errstr\n";

($time, $type)=&ora_fetch($db_list);

if ($time>178.25) {
if ( "Billed" eq $type ) {
$time=178.25;
}}

$totaled_hrs{$one_person[0]}+=$time;
$total+=$time;

}
############# end of perl ################

Result of java is
Hours: 154930.95 time all=5436908 time db=5297897 (sql=5434009) time
cycle=243005
Java=1.4.2_01

Which means 90 minutes for whole run.
Where 88 minutes for Oracle waiting.
Total Java expenses 4.5 minutes.
(Please, don't be too strict to numbers.
I just cut&paste them. Milliseconds are a little big time period.
Be indulgent to this quick and not precision way.)

perl result is:
Hours: 154930.95 for 1487 seconds.
Which is 25 minutes today.
Which is 3.6 times faster.

So, I'm open to any suggestions how to improve my Java code
or how to force managers to use Java in this project.

Alex Kizub.
 
D

dar7yl

Robert Klemme said:
Apparently not. I suggest you post code that compiles and is properly
indented.

What is your definition of "properly indented"?
Do you mean that hideous "K&R" standard of placing the brace
at the end of the line?

void myMethodIsBetterThanYours()
// a light-hearted dig at the style wars...
{
if ( youThinkYoursIsBetter() )
{
// flame wars begin ...
}
}
regards,
Dar7yl.
 
A

akizub

Dar7yl:
Thank for answer.
If your jdbc driver supports it, turn on tracing/profiling.
I use jodbc14.jar from Oracle. Don't know how to force it to trace.
Tomorrow will have a look on oracle.com.
But what is the diffence where time is spent? I still have to pay the
price.
if (rs.next())
{
// logic goes here
}
Of course I usually do this. You can see it from previous code.
I only tried to simplify the code and reduce number of getMilliseconds
calls.
Perhaps, you could optimize the query to return the whole list in one
call.

Of course I do it in real project.
And I can do it in both Java or perl.
But then quiry eats so much power of Oracle that DBAs call me and ask
to stop it.
Did you ever had such experience when your query runs for hour and DBA
call you and ask what is that query which combines 9 tables and hold
whole system for an hour?
What can I answer for such DBA? That this is your database structure
and your relations? And there is no other way to obtain data from your
imparctical design?
And people like this design and support this system...
So, I ask no questions.
I ask them here :))))

Anyway. It seems for me more and more that difference is in OCI driver
for perl. I'll try to play with it but will take time.
And anyway again it's rear approach. Most applications use thin driver.
And when it three times slower then OCI... too bad.

Thanks anyway. I already gave up.
But it was good experience for me.
I know a lot of new stuff and will know even more (with this OCI I
hope).

And project remains on perl. But who cares!
perl is much better then COBOL.
Another of my previous projects is still on COBOL :(((

Alex Kizub.
 
T

Tom Dyess

Dar7yl:
Thank for answer.
I use jodbc14.jar from Oracle. Don't know how to force it to trace.
Tomorrow will have a look on oracle.com.
But what is the diffence where time is spent? I still have to pay the
price.

Of course I usually do this. You can see it from previous code.
I only tried to simplify the code and reduce number of getMilliseconds
calls.

call.

Of course I do it in real project.
And I can do it in both Java or perl.
But then quiry eats so much power of Oracle that DBAs call me and ask
to stop it.
Did you ever had such experience when your query runs for hour and DBA
call you and ask what is that query which combines 9 tables and hold
whole system for an hour?
What can I answer for such DBA? That this is your database structure
and your relations? And there is no other way to obtain data from your
imparctical design?
And people like this design and support this system...
So, I ask no questions.
I ask them here :))))

Anyway. It seems for me more and more that difference is in OCI driver
for perl. I'll try to play with it but will take time.
And anyway again it's rear approach. Most applications use thin driver.
And when it three times slower then OCI... too bad.

Thanks anyway. I already gave up.
But it was good experience for me.
I know a lot of new stuff and will know even more (with this OCI I
hope).

And project remains on perl. But who cares!
perl is much better then COBOL.
Another of my previous projects is still on COBOL :(((

Alex Kizub.

Can't you exp the schema/tables and imp it onto another server and tune it
without bothering anyone else? Don't you have a development box at work?
There is something really wrong going on and you're gonna have to tinker
with it. One of us could probably find it after a few minutes if we could
connect to your environment, but in newsgroups, its just conjecture and
theory. Try putting a /*+ RULE */ hint in your SQL (after the first select)
and make sure your indexes are in place. You're connected to the proper
instance - i.e. there aren't two instances on the same box and you're
connecting to one via perl and the other via Java are you?
 
A

akizub

Can't you exp the schema/tables and imp it onto another server and
tune it
without bothering anyone else?
No. On devl it works fine.

Don't you have a development box at work?
Our DBAs can't replicate situation in devl. It exists only in prod.
There is something really wrong going on
I totally agree.
and you're gonna have to tinker with it.
And that's what I'm doing.
One of us could probably find it after a few minutes if we could
connect to your environment,
It's not even mine.
its just conjecture and theory.
And that's what I'm talking about.
Try putting a /*+ RULE */ hint in your SQL (after the first select)
and make sure your indexes are in place.
This is not about tune or performance. This is about comparisond two
languages.
So, everything should be the same. Bad or good.
You're connected to the proper
instance - i.e. there aren't two instances on the same box and you're
connecting to one via perl and the other via Java are you?

Are you kidding? Everything is the same.

Sorry. Couldn't resist to answer to guy from OraclePower.com
Even topic is closed.
Sincerely,
Alex Kizub.
 
T

Tom Dyess

tune it
without bothering anyone else?
No. On devl it works fine.


Our DBAs can't replicate situation in devl. It exists only in prod.

I totally agree.

And that's what I'm doing.

connect to your environment,
It's not even mine.

And that's what I'm talking about.

and make sure your indexes are in place.
This is not about tune or performance. This is about comparisond two
languages.
So, everything should be the same. Bad or good.

instance - i.e. there aren't two instances on the same box and you're
connecting to one via perl and the other via Java are you?

Are you kidding? Everything is the same.

Sorry. Couldn't resist to answer to guy from OraclePower.com
Even topic is closed.
Sincerely,
Alex Kizub.

Yea, I just don't like seeing it die. I like to solve problems but looks
like this one is taking a dirt nap. I thought the objective was to make the
Java version run at a comperable speed to Perl as opposed to more of an
academic nature. To do that, it requires more than your environment is
willing to let you do. Sounds like your DBA won't even let you work on it
much less give you explain plans, so you're kinda feeling in the dark. Oh
well, it was fun anyway.

Tom Dyess
OraclePower.com
 
?

=?iso-8859-1?Q?Robert_Sundstr=F6m?=

Luke Webber wrote on 01.02.2005 22:49:

I don't think that's a good idea. That will leave him with a lot of
unclosed cursors...

Not necessarily.

If I understand things, the original poster has an iteration where he
executes a PreparedStatement over and over again. Any open ResultSet
objects originating from that PreparedStatement should, if the JDBC driver
complies with the JDBC spec, be automatically closed at the next execute.

However, it would be most strange if the 2000 extra round trips to the
server can cause the delay. I would say that the server, for some reason
optimizes the SQL-statements differently. It could be that the Oracle-Perl
driver internally flips some magic option or compilation directive which
in turn makes the optimizer make better or less good decisions. The
original poster is probably better off asking this question at an Oracle
group.
 
D

Dimitri Maziuk

Thomas Kellerer sez:
Luke Webber wrote on 01.02.2005 22:49:

I don't think that's a good idea. That will leave him with a lot of
unclosed cursors...

Nope. Oracle doesn't close the cursor on rs.close(), it closes on
query.close() (by default, anyway).

Dima
 
V

Virgil Green

tune it
without bothering anyone else?
No. On devl it works fine.

Someone correct me if I'm wrong, but isn't this the first mention of there
being a Development environment in which the problem does not exist?

Perl and Java provide similar performance in Development, but Java lags
significantly in Production?

Sounds like a configuration problem.
 
A

Alex

My apology. I became victim of circumstances and my own ambitions.
That's production database and performance varies too much.
Right now I did a lot of other statistics and discovered that in
average SQL time varies 100 times.
So, when I saw that perl was faster (and it was a little faster because
used OCI driver) and after 3 to 5 runs (remember, they take about hour)
I changed SQL from plain one to prepared statement. So, when perl sent
new SQL every time Java used parameters in predefined one. And it
became even slower.
I was upset and started this topic.

Now DBA (other, not ours) did a little research and discovered that
Explanation Plan which Oracle uses for prepared statement is totally
different then for plain text SQL. And based on whatever it was based
this plan in reality was much worse that plan for the same SQL but with
values except parameters.

So, I changed SQL back from prepared to generated for each run (just as
perl did) and got approximately the same results for both languages.
Both, again, were different time to time at least 10 times but, at
least similar. Sometime java average time was from 10 to 400 seconds
for 100 SQL, sometime perl average time was from 500 to 10. Looks
pretty unstable but on the same level.

Conclusion is obvious. Use development environment, have more
statistics, have control, work, work and work...

Thanks everybody. Java is the best. As well as perl. And C++. Not C# or
..NET!
Alex Kizub.
 
C

Chris Smith

Alex said:
So, I changed SQL back from prepared to generated for each run (just as
perl did) and got approximately the same results for both languages.

I can't help but point out how much time you spent dismissing and
berating people on this newsgroup who tried to tell you that something
like that was going on. Maybe you can change your attitude and behavior
in the future.

--
www.designacourse.com
The Easiest Way To Train Anyone... Anywhere.

Chris Smith - Lead Software Developer/Technical Trainer
MindIQ Corporation
 
L

Luke Webber

Alex said:
My apology. I became victim of circumstances and my own ambitions.
That's production database and performance varies too much.
Right now I did a lot of other statistics and discovered that in
average SQL time varies 100 times.
So, when I saw that perl was faster (and it was a little faster because
used OCI driver) and after 3 to 5 runs (remember, they take about hour)
I changed SQL from plain one to prepared statement. So, when perl sent
new SQL every time Java used parameters in predefined one. And it
became even slower.
I was upset and started this topic.

Now DBA (other, not ours) did a little research and discovered that
Explanation Plan which Oracle uses for prepared statement is totally
different then for plain text SQL. And based on whatever it was based
this plan in reality was much worse that plan for the same SQL but with
values except parameters.
[snip]

Well thank goodness for that! Glad you got it sorted out
Conclusion is obvious. Use development environment, have more
statistics, have control, work, work and work...

I'll add one more conclusion. When in doubt, proceed by steps, and when
one step doesn't help, back it out. Piling change upon change is
unhelpful, and will just leave you floundering.

Now if only I could learn to take my own advice. said:
Thanks everybody. Java is the best. As well as perl. And C++. Not C# or
.NET!

Worthy sentiments. ;^)

Luke
 
S

steve

Here is code. Exactly. Except the SQL.
Hashtable ht=new Hashtable();
PreparedStatement ps =
conection.getPreparedStatement("select 'key', sysdate+? from
dual");

for (int i=0; i<2000; i++)
ps.setInt(1, i);

rs = ps.executeQuery();
if (rs.next()) {
String key=rs.getString(1);
double value=0;
Double d=(Double)ht.get(key);
if (d!=null) value=d.doubleValue();
value+=rs.getDouble(2);
ht.put(key,new Double(value));
}
rs.close();
}

That's all. Exceptions are outside and handled once. No existing null
values.
Hashtable is 1000 or less keys. Not big deal. As well as possible
impovements of code like mutable Objects instead Double or similar.

And that's all! No Unicode expenses or huge network trafic.
Thin driver.
I'll try OCI driver. Always was sure that thin is good enough.
Keep guessing! Because it's the shock for me.

Alex Kizub.

loose the first "if" & replace with a while.
loose the checks for types inside you loop, replace it with a getobject.

that should get you as fast as you are going to be using the thin driver.,
using the above code.

but you need to rewrite the whole thing, why loop round a prepared statement
2000 times, that is just madness.

try the following system :


write the code as a serverside pl/sql routine that returns a cursor of 2,000
cols, then call the routine.
it will reduce your network calls by atleast 50%, and reduce the servers shit
down even further.

but i cannot see why you have to do what you are doing , the way you do it.



steve
 
S

steve

Looks like this for me now :(((

allow
you to bench mark

See this is production and, actually, third party application.
Do you think they will install something new for Java when perl is
already faster?

no but it wil lallow you to find out the fastest you are EVER going to run.
Don't understand this. I tried few different JDBC thin drivers
including latest Oracle ojdbc14.jar for our particlular Oracle9i
9.2.0.4 JDBC Drivers.

sorry!!
the jdbc has 2 modes.
1. pure java.
2. java callouts to c++ oci system.

it depends on how you call the java driver when you first make hte
connection.
See another anwers in this thraed. Pretty simple.
Here is code. Exactly. Except the SQL.
Hashtable ht=new Hashtable();
PreparedStatement ps =
conection.getPreparedStatement("select 'key', sysdate+? from
dual");

for (int i=0; i<2000; i++)
ps.setInt(1, i);

rs = ps.executeQuery();
if (rs.next()) {
String key=rs.getString(1);
double value=0;
Double d=(Double)ht.get(key);
if (d!=null) value=d.doubleValue();
value+=rs.getDouble(2);
ht.put(key,new Double(value));

}
rs.close();
}

Still have a little hope. But less and less...
Alex Kizub.

see my post else where for problems on your code.

steve
 
S

steve

It seems clear that you're correct about the time being spent in the
JDBC connection logic, but I'm wondering just which part of the JDBC
code is taking all the time. The figures aren't fine-grained enough.

I have one off-the-wall suggestion. Why not try removing the
"rs.close()" statement and see how it goes? I have a niggling feeling
that it might be making an extra round trip to the database server for
some reason.

Cheers,
Luke

he should not be calling an sql statement 2000 times, when he can do it in
the server, and return a cursor.

and if he is so stubbon to do a callout 2000 times he can bulk it up to 1
call, execute & then close.
not callout 2,000 times , closing each time.

the code is broken, implementation wise, and it can be profiled to the end of
time, but it is not going to be able to fix it like that.

steve
 
S

steve

It seems clear that you're correct about the time being spent in the
JDBC connection logic, but I'm wondering just which part of the JDBC
code is taking all the time. The figures aren't fine-grained enough.

I have one off-the-wall suggestion. Why not try removing the
"rs.close()" statement and see how it goes? I have a niggling feeling
that it might be making an extra round trip to the database server for
some reason.

Cheers,
Luke

it will stop ( well actually sql crash , and it would kill any other program
on the server that tries to open a cursor ) after 300 cursors are left open.
not the way to run on a production system.

steve
 
C

Chris Smith

steve said:
it will stop ( well actually sql crash , and it would kill any other program
on the server that tries to open a cursor ) after 300 cursors are left open.
not the way to run on a production system.

Steve, you're a little late here, but just to clarify: that's not true.
Omitting rs.close() in the code mentioned here will not leave open
cursors anywhere, because the cursor will be closed when the next loop
iteration calls PreparedStatement.execute(). The last cursor is closed
when the connection is closed.

It is possible to avoid this behavior, using the prepareStatement
overload (new in 1.4) that takes an additional parameter on result set
holdability. Since this is not used in the OP's code, your concern is
not valid. It turns out that there's very rarely a reason to explicitly
close ResultSet instances, unless you went out of your way to keep them
open in the first place.

In any case, Alex already has a solution to his problem.

--
www.designacourse.com
The Easiest Way To Train Anyone... Anywhere.

Chris Smith - Lead Software Developer/Technical Trainer
MindIQ Corporation
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,770
Messages
2,569,583
Members
45,072
Latest member
trafficcone

Latest Threads

Top