code to large for machine-generated code

[Thomas Richter]

Hi folks,

is there some way how to persuade the javac compiler to accept very long
methods? No, don't worry, I'm not writing this kind of mess. The code in
question is the result of a meta-compilation from another language, and
it turned out that this compiler generated a pretty long java code from
a seemingly simple source. Unfortunately, the compilation of the
generated java code then fails with the infamous "code too large" error.

Is there any kind of tweaking that can be done to make this code
acceptable (besides fixing the compiler that generated the java code
in first place, that is.)?

So long,
Thomas

 

[bugbear]

Hi folks,

is there some way how to persuade the javac compiler to accept very long
methods? No, don't worry, I'm not writing this kind of mess. The code in
question is the result of a meta-compilation from another language, and
it turned out that this compiler generated a pretty long java code from
a seemingly simple source. Unfortunately, the compilation of the
generated java code then fails with the infamous "code too large" error.

Is there any kind of tweaking that can be done to make this code
acceptable (besides fixing the compiler that generated the java code
in first place, that is.)?

Turn off optional optimisations.

BugBear

 

[Chris Uppal]

is there some way how to persuade the javac compiler to accept very long
methods?
[...]
Is there any kind of tweaking that can be done to make this code
acceptable (besides fixing the compiler that generated the java code
in first place, that is.)?

'fraid not. The limit is built into the classfile format, and won't change
until that changes

There was some hope of that happening in 1.5, but it didn't materialise. As
far as I can tell, despite the fact that the classfile format will change in
1.6, none of the changes will remove the limits on the lengths of compiled
methods (or related limits).

-- chris

 

[Roedy Green]

Unfortunately, the compilation of the
generated java code then fails with the infamous "code too large" error.

If you code is too large, the problem is the JVM byte code addressing
can't span the method. You need to do something to make the generated
code shorter or to split it up. See what you could pull off with some
anonymous classes. I would disassemble the code to see if you can see
any patterns you might encapsulate or code that is inlining
excessively long.

see http://mindprod.com/jgloss/disassembler.html

 

[jmcgill]

Hi folks,

is there some way how to persuade the javac compiler to accept very long
methods? No, don't worry, I'm not writing this kind of mess. The code in
question is the result of a meta-compilation from another language, and
it turned out that this compiler generated a pretty long java code from
a seemingly simple source.

The fact is your code generator is violating a requirement of the target
language.

My suspicion is that the method-size limit exists in order to facilitate
one or a few of the runtime target platforms, but that's neither here
nor there. It's a language spec and your code generator is violating
it, and the compiler is rejecting it as it would any other fatal error.

What are you doing? Setting up a static array from a data structure
that represents an image, or something like that?

For the record, I think the 64K limitation is stupid, and it makes me
wonder if we're shackled with JVM support for 16-bit address registers
somewhere.

I'm told that you can have larger than 64K methods by eliminating try
and synchronized blocks, and compiling without line number information.

This is not the only 16-bit limitation in the classfile format; there
are actually quite a few. But where nobody is likely to have 65536
local variable in scope, or to put 256 parameters on a method call stack
(or 128 longs!), or to name a field with more than 65535 characters, we
certainly have seen 65536+ byte methods, at least from data definition
blocks.

Do you actually have a procedural method that gets this limit? Sounds
like your source material is pretty nasty itself.

 

[Mike Schilling]

Hi folks,

is there some way how to persuade the javac compiler to accept very long
methods? No, don't worry, I'm not writing this kind of mess. The code in
question is the result of a meta-compilation from another language, and it
turned out that this compiler generated a pretty long java code from
a seemingly simple source. Unfortunately, the compilation of the generated
java code then fails with the infamous "code too large" error.

Is there any kind of tweaking that can be done to make this code
acceptable (besides fixing the compiler that generated the java code
in first place, that is.)?

The problem is, I'm guessing, single methods generating bytecode that's too
long. If you look at the generated Java, you might find patterns that lend
themselves to a mechanical fix for this, so that you could post-process the
generated Java instead of changing the generator. If you could recognize
long stretches that share a fairly small set of local variables with the
rest of the code, it should be straightforward to move them to private
methods.


I've actually worked with a real-life example of this, a YACC compiler that
generated switches too large for the C++ compiler we were using. In that
case, we had the YACC source and could modify it, but a post-processor would
have worked equally well. The fix was to change

switch (i)
{
case 1: ...
case 2:...
...
case 10000:...
}

to

if (i < 1000)
{
case 1:...
...
case 999:...
}
else if (i < 2000)
{
...
}
etc.

 

[Chris Uppal]

I'm told that you can have larger than 64K methods by eliminating try
and synchronized blocks, and compiling without line number information.

I've tried it; the verifier rejects it even if it would otherwise be a
structurally valid classfile.

(That was probably tested on 1.5.0, but might have been 1.4.2 -- I forget)

-- chris

 

[Roedy Green]

For the record, I think the 64K limitation is stupid, and it makes me
wonder if we're shackled with JVM support for 16-bit address registers
somewhere.

It is a side effect of the class file format. They wanted to make it
compact.
See http://mindprod.com/jgloss/javaclassfileformat.html

 

[Luc The Perverse]

If you code is too large, the problem is the JVM byte code addressing
can't span the method. You need to do something to make the generated
code shorter or to split it up. See what you could pull off with some
anonymous classes. I would disassemble the code to see if you can see
any patterns you might encapsulate or code that is inlining
excessively long.

see http://mindprod.com/jgloss/disassembler.html

I don't know how long this method is, but I have a gut instinct that says
way too much functionality is being crammed into a function, which probably
should have its own class with different several different methods.

Times when I have had astronomically large functions, I have always been
doing something stupid.

 

[Mike Schilling]

It is a side effect of the class file format. They wanted to make it
compact.
See http://mindprod.com/jgloss/javaclassfileformat.html

That link doesn;t work for me, but this one does:

http://mindprod.com/jgloss/javaclassformat.html

 

[Thomas Richter]

I don't know how long this method is, but I have a gut instinct that says
way too much functionality is being crammed into a function, which probably
should have its own class with different several different methods.

No, it's not. As said, *I* haven't written this code. The computer did.
(-; The source of all this is surprisingly small, around 20 lines.

Times when I have had astronomically large functions, I have always been
doing something stupid.

Unless you're not the one who's doing it right away in first place. (-;

So long,
Thomas

 

[Chris Uppal]

I don't know how long this method is, but I have a gut instinct that says
way too much functionality is being crammed into a function, which
probably should have its own class with different several different
methods.

I remind you of the context. From the original post:

No, don't worry, I'm not writing this kind of mess. The code in
question is the result of a meta-compilation from another language, and
it turned out that this compiler generated a pretty long java code from
a seemingly simple source.

-- chris

 

[Chris Uppal]

The source of all this is surprisingly small, around 20 lines.

It would be interesting to know what it is and why it expands so much. Is that
something you are allowed to discus ?

-- chris

 

[jmcgill]

No, it's not. As said, *I* haven't written this code. The computer did.
(-; The source of all this is surprisingly small, around 20 lines.

Really? I wonder if just one or two statements are causing the problem.
Sounds like a good thing to submit to the daily WTF?

What is the shortest valid java source that generates an invalid (due to
size constraints) byte code?

I was picturing an array initializer that sets up an audio or image
file, or else a series of conditionals or case statements or assignments
that ran long.

20 statements or so? That *is* interesting. Wish you would post it.

 

[Luc The Perverse]

Really? I wonder if just one or two statements are causing the problem.
Sounds like a good thing to submit to the daily WTF?

I wonder if he means the code which generates the [much longer] function is
only 20 lines long?

I did this once - I made a function which executed immediately calling
functions which appeared to execute functionality, but instead only
generated code, but it generated turn based iterative code which was thrown
into a gigantic switch statement.

 

[Oliver Wong]

Really? I wonder if just one or two statements are causing the problem.
Sounds like a good thing to submit to the daily WTF?

What is the shortest valid java source that generates an invalid (due to
size constraints) byte code?

I was picturing an array initializer that sets up an audio or image file,
or else a series of conditionals or case statements or assignments that
ran long.

20 statements or so? That *is* interesting. Wish you would post it.

From the OP:

<quote>
The code in
question is the result of a meta-compilation from another language, and
it turned out that this compiler generated a pretty long java code from
a seemingly simple source.
</quote>

I.e. the original code might be something like:

<someWeirdLanguage>
$B_halts = determines_if_program_halts "~/src/cpp/myComplex.cpp"
</someWeirdLanguage>

which turns into a Java program which solves the halting problem. In other
words, the source language isn't Java.

- Oliver

 

[jmcgill]

which turns into a Java program which solves the halting problem. In
other words, the source language isn't Java.

I see what you're saying. I don't care about the original source
language. Are you using some tongue in cheek irony there, e.g. "solves
The Halting Problem" ?


Through all this discussion of several days, the OP has not
characterized the java code that ends up producing invalid byte code.

If the java code is of a reasonable size but produces bytecode that is
not, that's interesting. But if the java code looks like:

public static void initializeBMP {
// many megabytes of this...
int[] bmpbytes = { 0x00, 0x00, ..., 0x2A };
}

well, that's not a reasonable thing to do.

I also stand by my first comment -- if a code generator outputs
something that's plainly invalid according to the target language spec,
it's broken.

But it's still not clear whether the problem is javac turning reasonable
java statements into invalid bytecode, or whether the problem is a code
translator that creates unreasonable (or invalid) java source.

The OP makes it sound like the java source looks okay. And if that's
the case, I'd certainly like to see it.

 

[Chris Uppal]

Through all this discussion of several days, the OP has not
characterized the java code that ends up producing invalid byte code.

Well, he hadn't been asked until today.

I also stand by my first comment -- if a code generator outputs
something that's plainly invalid according to the target language spec,
it's broken.

I may have forgotten something but I don't remember a limit on method size in
the Java Language Specification. (There are limits in classfiles, of course,
but that's a different issue.)

But it's still not clear whether the problem is javac turning reasonable
java statements into invalid bytecode, or whether the problem is a code
translator that creates unreasonable (or invalid) java source.

On the basic of what's been said so far (but I'd like to know more) it sounds
as if the code generator produces perfectly valid Java code, but code which
javac is unable to translate into JVM bytecode.

-- chris

 

[jmcgill]

I may have forgotten something but I don't remember a limit on method size in
the Java Language Specification. (There are limits in classfiles, of course,
but that's a different issue.)

The classfile format is the only issue!

There are several 16-bit and one or two 8-bit constraints in the
classfile spec, and my understanding is the method size in this
discussion is one of those.

4.10 Limitations of the Java Virtual Machine
....
* The amount of code per non-native, non-abstract method is limited to
65536 bytes by the sizes of the indices in the exception_table of the
Code attribute (§4.7.3), in the LineNumberTable attribute (§4.7.8), and
in the LocalVariableTable attribute (§4.7.9).

On the basic of what's been said so far (but I'd like to know more) it sounds
as if the code generator produces perfectly valid Java code, but code which
javac is unable to translate into JVM bytecode.

That's what makes it interesting, and I suspect, in the category of
"javac bug."

Of course it's difficult or impossible to determine from a source file
whether the compiler output from that source will be valid. But it
would be very interesting to see some reasonable looking routine that
turns into some brokenly huge bytecode.


The compiler should know when it's writing an invalid classfile and
should abort with a fatal error.

 

[Oliver Wong]

From my understanding, the OP has some compiler which takes source code
written into some language (call it language A), and produces code written
in another language (language B). It just so happens that language B is a
superset of Java. I.e., it is exactly like Java, but without various size
limitations.

The OP is then taking this program, written in language B, and giving it
to javac, hoping to fool javac into believing that this is actually a
program written in Java. Usually this works, but the OP ran into this one
instance where it doesn't work.

- Oliver