code to large for machine-generated code

Discussion in 'Java' started by Thomas Richter, May 10, 2006.

  1. Hi folks,

    is there some way how to persuade the javac compiler to accept very long
    methods? No, don't worry, I'm not writing this kind of mess. The code in
    question is the result of a meta-compilation from another language, and
    it turned out that this compiler generated a pretty long java code from
    a seemingly simple source. Unfortunately, the compilation of the
    generated java code then fails with the infamous "code too large" error.

    Is there any kind of tweaking that can be done to make this code
    acceptable (besides fixing the compiler that generated the java code
    in first place, that is.)?

    So long,
    Thomas
    Thomas Richter, May 10, 2006
    #1
    1. Advertising

  2. Thomas Richter

    bugbear Guest

    Thomas Richter wrote:
    > Hi folks,
    >
    > is there some way how to persuade the javac compiler to accept very long
    > methods? No, don't worry, I'm not writing this kind of mess. The code in
    > question is the result of a meta-compilation from another language, and
    > it turned out that this compiler generated a pretty long java code from
    > a seemingly simple source. Unfortunately, the compilation of the
    > generated java code then fails with the infamous "code too large" error.
    >
    > Is there any kind of tweaking that can be done to make this code
    > acceptable (besides fixing the compiler that generated the java code
    > in first place, that is.)?


    Turn off optional optimisations.

    BugBear
    bugbear, May 10, 2006
    #2
    1. Advertising

  3. Thomas Richter

    Chris Uppal Guest

    Thomas Richter wrote:

    > is there some way how to persuade the javac compiler to accept very long
    > methods?
    > [...]
    > Is there any kind of tweaking that can be done to make this code
    > acceptable (besides fixing the compiler that generated the java code
    > in first place, that is.)?


    'fraid not. The limit is built into the classfile format, and won't change
    until that changes

    There was some hope of that happening in 1.5, but it didn't materialise. As
    far as I can tell, despite the fact that the classfile format will change in
    1.6, none of the changes will remove the limits on the lengths of compiled
    methods (or related limits).

    -- chris
    Chris Uppal, May 10, 2006
    #3
  4. Thomas Richter

    Roedy Green Guest

    On Wed, 10 May 2006 13:58:36 +0200, Thomas Richter
    <-Berlin.DE> wrote, quoted or indirectly quoted someone
    who said :

    > Unfortunately, the compilation of the
    >generated java code then fails with the infamous "code too large" error.


    If you code is too large, the problem is the JVM byte code addressing
    can't span the method. You need to do something to make the generated
    code shorter or to split it up. See what you could pull off with some
    anonymous classes. I would disassemble the code to see if you can see
    any patterns you might encapsulate or code that is inlining
    excessively long.

    see http://mindprod.com/jgloss/disassembler.html

    --
    Canadian Mind Products, Roedy Green.
    http://mindprod.com Java custom programming, consulting and coaching.
    Roedy Green, May 10, 2006
    #4
  5. Thomas Richter

    jmcgill Guest

    Thomas Richter wrote:
    > Hi folks,
    >
    > is there some way how to persuade the javac compiler to accept very long
    > methods? No, don't worry, I'm not writing this kind of mess. The code in
    > question is the result of a meta-compilation from another language, and
    > it turned out that this compiler generated a pretty long java code from
    > a seemingly simple source.


    The fact is your code generator is violating a requirement of the target
    language.

    My suspicion is that the method-size limit exists in order to facilitate
    one or a few of the runtime target platforms, but that's neither here
    nor there. It's a language spec and your code generator is violating
    it, and the compiler is rejecting it as it would any other fatal error.

    What are you doing? Setting up a static array from a data structure
    that represents an image, or something like that?

    For the record, I think the 64K limitation is stupid, and it makes me
    wonder if we're shackled with JVM support for 16-bit address registers
    somewhere.

    I'm told that you can have larger than 64K methods by eliminating try
    and synchronized blocks, and compiling without line number information.

    This is not the only 16-bit limitation in the classfile format; there
    are actually quite a few. But where nobody is likely to have 65536
    local variable in scope, or to put 256 parameters on a method call stack
    (or 128 longs!), or to name a field with more than 65535 characters, we
    certainly have seen 65536+ byte methods, at least from data definition
    blocks.

    Do you actually have a procedural method that gets this limit? Sounds
    like your source material is pretty nasty itself.
    jmcgill, May 10, 2006
    #5
  6. "Thomas Richter" <-Berlin.DE> wrote in message
    news:...
    > Hi folks,
    >
    > is there some way how to persuade the javac compiler to accept very long
    > methods? No, don't worry, I'm not writing this kind of mess. The code in
    > question is the result of a meta-compilation from another language, and it
    > turned out that this compiler generated a pretty long java code from
    > a seemingly simple source. Unfortunately, the compilation of the generated
    > java code then fails with the infamous "code too large" error.
    >
    > Is there any kind of tweaking that can be done to make this code
    > acceptable (besides fixing the compiler that generated the java code
    > in first place, that is.)?


    The problem is, I'm guessing, single methods generating bytecode that's too
    long. If you look at the generated Java, you might find patterns that lend
    themselves to a mechanical fix for this, so that you could post-process the
    generated Java instead of changing the generator. If you could recognize
    long stretches that share a fairly small set of local variables with the
    rest of the code, it should be straightforward to move them to private
    methods.


    I've actually worked with a real-life example of this, a YACC compiler that
    generated switches too large for the C++ compiler we were using. In that
    case, we had the YACC source and could modify it, but a post-processor would
    have worked equally well. The fix was to change

    switch (i)
    {
    case 1: ...
    case 2:...
    ...
    case 10000:...
    }

    to

    if (i < 1000)
    {
    case 1:...
    ...
    case 999:...
    }
    else if (i < 2000)
    {
    ...
    }
    etc.
    Mike Schilling, May 10, 2006
    #6
  7. Thomas Richter

    Chris Uppal Guest

    jmcgill wrote:

    > I'm told that you can have larger than 64K methods by eliminating try
    > and synchronized blocks, and compiling without line number information.


    I've tried it; the verifier rejects it even if it would otherwise be a
    structurally valid classfile.

    (That was probably tested on 1.5.0, but might have been 1.4.2 -- I forget)

    -- chris
    Chris Uppal, May 10, 2006
    #7
  8. Thomas Richter

    Roedy Green Guest

    On Wed, 10 May 2006 11:38:48 -0700, jmcgill
    <> wrote, quoted or indirectly quoted someone
    who said :

    >For the record, I think the 64K limitation is stupid, and it makes me
    >wonder if we're shackled with JVM support for 16-bit address registers
    >somewhere.


    It is a side effect of the class file format. They wanted to make it
    compact.
    See http://mindprod.com/jgloss/javaclassfileformat.html
    --
    Canadian Mind Products, Roedy Green.
    http://mindprod.com Java custom programming, consulting and coaching.
    Roedy Green, May 10, 2006
    #8
  9. "Roedy Green" <> wrote in
    message news:...
    > On Wed, 10 May 2006 13:58:36 +0200, Thomas Richter
    > <-Berlin.DE> wrote, quoted or indirectly quoted someone
    > who said :
    >
    >> Unfortunately, the compilation of the
    >>generated java code then fails with the infamous "code too large" error.

    >
    > If you code is too large, the problem is the JVM byte code addressing
    > can't span the method. You need to do something to make the generated
    > code shorter or to split it up. See what you could pull off with some
    > anonymous classes. I would disassemble the code to see if you can see
    > any patterns you might encapsulate or code that is inlining
    > excessively long.
    >
    > see http://mindprod.com/jgloss/disassembler.html


    I don't know how long this method is, but I have a gut instinct that says
    way too much functionality is being crammed into a function, which probably
    should have its own class with different several different methods.

    Times when I have had astronomically large functions, I have always been
    doing something stupid.

    --
    LTP

    :)
    Luc The Perverse, May 10, 2006
    #9
  10. "Roedy Green" <> wrote in
    message news:...
    > On Wed, 10 May 2006 11:38:48 -0700, jmcgill
    > <> wrote, quoted or indirectly quoted someone
    > who said :
    >
    >>For the record, I think the 64K limitation is stupid, and it makes me
    >>wonder if we're shackled with JVM support for 16-bit address registers
    >>somewhere.

    >
    > It is a side effect of the class file format. They wanted to make it
    > compact.
    > See http://mindprod.com/jgloss/javaclassfileformat.html


    That link doesn;t work for me, but this one does:

    http://mindprod.com/jgloss/javaclassformat.html
    Mike Schilling, May 10, 2006
    #10
  11. Luc The Perverse wrote:

    > I don't know how long this method is, but I have a gut instinct that says
    > way too much functionality is being crammed into a function, which probably
    > should have its own class with different several different methods.


    No, it's not. As said, *I* haven't written this code. The computer did.
    (-; The source of all this is surprisingly small, around 20 lines.

    > Times when I have had astronomically large functions, I have always been
    > doing something stupid.


    Unless you're not the one who's doing it right away in first place. (-;

    So long,
    Thomas
    Thomas Richter, May 11, 2006
    #11
  12. Thomas Richter

    Chris Uppal Guest

    Luc The Perverse wrote:

    > I don't know how long this method is, but I have a gut instinct that says
    > way too much functionality is being crammed into a function, which
    > probably should have its own class with different several different
    > methods.


    I remind you of the context. From the original post:

    > No, don't worry, I'm not writing this kind of mess. The code in
    > question is the result of a meta-compilation from another language, and
    > it turned out that this compiler generated a pretty long java code from
    > a seemingly simple source.


    -- chris
    Chris Uppal, May 11, 2006
    #12
  13. Thomas Richter

    Chris Uppal Guest

    Thomas Richter wrote:

    > The source of all this is surprisingly small, around 20 lines.


    It would be interesting to know what it is and why it expands so much. Is that
    something you are allowed to discus ?

    -- chris
    Chris Uppal, May 11, 2006
    #13
  14. Thomas Richter

    jmcgill Guest

    Thomas Richter wrote:

    > No, it's not. As said, *I* haven't written this code. The computer did.
    > (-; The source of all this is surprisingly small, around 20 lines.


    Really? I wonder if just one or two statements are causing the problem.
    Sounds like a good thing to submit to the daily WTF?

    What is the shortest valid java source that generates an invalid (due to
    size constraints) byte code?

    I was picturing an array initializer that sets up an audio or image
    file, or else a series of conditionals or case statements or assignments
    that ran long.

    20 statements or so? That *is* interesting. Wish you would post it.
    jmcgill, May 11, 2006
    #14
  15. "jmcgill" <> wrote in message
    news:...
    > Thomas Richter wrote:
    >
    >> No, it's not. As said, *I* haven't written this code. The computer did.
    >> (-; The source of all this is surprisingly small, around 20 lines.

    >
    > Really? I wonder if just one or two statements are causing the problem.
    > Sounds like a good thing to submit to the daily WTF?


    I wonder if he means the code which generates the [much longer] function is
    only 20 lines long?

    I did this once - I made a function which executed immediately calling
    functions which appeared to execute functionality, but instead only
    generated code, but it generated turn based iterative code which was thrown
    into a gigantic switch statement.

    --
    LTP

    :)
    Luc The Perverse, May 11, 2006
    #15
  16. Thomas Richter

    Oliver Wong Guest

    "jmcgill" <> wrote in message
    news:...
    > Thomas Richter wrote:
    >
    >> No, it's not. As said, *I* haven't written this code. The computer did.
    >> (-; The source of all this is surprisingly small, around 20 lines.

    >
    > Really? I wonder if just one or two statements are causing the problem.
    > Sounds like a good thing to submit to the daily WTF?
    >
    > What is the shortest valid java source that generates an invalid (due to
    > size constraints) byte code?
    >
    > I was picturing an array initializer that sets up an audio or image file,
    > or else a series of conditionals or case statements or assignments that
    > ran long.
    >
    > 20 statements or so? That *is* interesting. Wish you would post it.


    From the OP:

    <quote>
    The code in
    question is the result of a meta-compilation from another language, and
    it turned out that this compiler generated a pretty long java code from
    a seemingly simple source.
    </quote>

    I.e. the original code might be something like:

    <someWeirdLanguage>
    $B_halts = determines_if_program_halts "~/src/cpp/myComplex.cpp"
    </someWeirdLanguage>

    which turns into a Java program which solves the halting problem. In other
    words, the source language isn't Java.

    - Oliver
    Oliver Wong, May 11, 2006
    #16
  17. Thomas Richter

    jmcgill Guest

    Oliver Wong wrote:

    > which turns into a Java program which solves the halting problem. In
    > other words, the source language isn't Java.


    I see what you're saying. I don't care about the original source
    language. Are you using some tongue in cheek irony there, e.g. "solves
    The Halting Problem" ?


    Through all this discussion of several days, the OP has not
    characterized the java code that ends up producing invalid byte code.

    If the java code is of a reasonable size but produces bytecode that is
    not, that's interesting. But if the java code looks like:

    public static void initializeBMP {
    // many megabytes of this...
    int[] bmpbytes = { 0x00, 0x00, ..., 0x2A };
    }

    well, that's not a reasonable thing to do.

    I also stand by my first comment -- if a code generator outputs
    something that's plainly invalid according to the target language spec,
    it's broken.

    But it's still not clear whether the problem is javac turning reasonable
    java statements into invalid bytecode, or whether the problem is a code
    translator that creates unreasonable (or invalid) java source.

    The OP makes it sound like the java source looks okay. And if that's
    the case, I'd certainly like to see it.
    jmcgill, May 11, 2006
    #17
  18. Thomas Richter

    Chris Uppal Guest

    jmcgill wrote:

    > Through all this discussion of several days, the OP has not
    > characterized the java code that ends up producing invalid byte code.


    Well, he hadn't been asked until today.


    > I also stand by my first comment -- if a code generator outputs
    > something that's plainly invalid according to the target language spec,
    > it's broken.


    I may have forgotten something but I don't remember a limit on method size in
    the Java Language Specification. (There are limits in classfiles, of course,
    but that's a different issue.)


    > But it's still not clear whether the problem is javac turning reasonable
    > java statements into invalid bytecode, or whether the problem is a code
    > translator that creates unreasonable (or invalid) java source.


    On the basic of what's been said so far (but I'd like to know more) it sounds
    as if the code generator produces perfectly valid Java code, but code which
    javac is unable to translate into JVM bytecode.

    -- chris
    Chris Uppal, May 11, 2006
    #18
  19. Thomas Richter

    jmcgill Guest

    Chris Uppal wrote:
    > I may have forgotten something but I don't remember a limit on method size in
    > the Java Language Specification. (There are limits in classfiles, of course,
    > but that's a different issue.)


    The classfile format is the only issue!

    There are several 16-bit and one or two 8-bit constraints in the
    classfile spec, and my understanding is the method size in this
    discussion is one of those.

    4.10 Limitations of the Java Virtual Machine
    ....
    * The amount of code per non-native, non-abstract method is limited to
    65536 bytes by the sizes of the indices in the exception_table of the
    Code attribute (§4.7.3), in the LineNumberTable attribute (§4.7.8), and
    in the LocalVariableTable attribute (§4.7.9).

    > On the basic of what's been said so far (but I'd like to know more) it sounds
    > as if the code generator produces perfectly valid Java code, but code which
    > javac is unable to translate into JVM bytecode.


    That's what makes it interesting, and I suspect, in the category of
    "javac bug."

    Of course it's difficult or impossible to determine from a source file
    whether the compiler output from that source will be valid. But it
    would be very interesting to see some reasonable looking routine that
    turns into some brokenly huge bytecode.


    The compiler should know when it's writing an invalid classfile and
    should abort with a fatal error.
    jmcgill, May 11, 2006
    #19
  20. Thomas Richter

    Oliver Wong Guest

    "Chris Uppal" <-THIS.org> wrote in message
    news:44638c8d$2$652$...
    > jmcgill wrote:
    >
    >> I also stand by my first comment -- if a code generator outputs
    >> something that's plainly invalid according to the target language spec,
    >> it's broken.


    From my understanding, the OP has some compiler which takes source code
    written into some language (call it language A), and produces code written
    in another language (language B). It just so happens that language B is a
    superset of Java. I.e., it is exactly like Java, but without various size
    limitations.

    The OP is then taking this program, written in language B, and giving it
    to javac, hoping to fool javac into believing that this is actually a
    program written in Java. Usually this works, but the OP ran into this one
    instance where it doesn't work.

    - Oliver
    Oliver Wong, May 11, 2006
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?UmVzaG1hIFByYWJodQ==?=

    Auto-attach to process '[1084] aspnet_wp.exe' on machine <Machine

    =?Utf-8?B?UmVzaG1hIFByYWJodQ==?=, Jan 21, 2005, in forum: ASP .Net
    Replies:
    0
    Views:
    1,053
    =?Utf-8?B?UmVzaG1hIFByYWJodQ==?=
    Jan 21, 2005
  2. UJ
    Replies:
    2
    Views:
    423
  3. Karsten Wutzke
    Replies:
    2
    Views:
    1,939
    Roedy Green
    Nov 4, 2003
  4. Replies:
    2
    Views:
    431
    Jerry Coffin
    Jan 14, 2008
  5. Dan Stromberg
    Replies:
    0
    Views:
    456
    Dan Stromberg
    Dec 23, 2010
Loading...

Share This Page