New language on JVM?

[jamesl]

Hi All,

If you have ever wanted to implement a new language on top of the JVM,
then this series may be of
interest to you: Redline: Smalltalk in 'small talks' ....
http://blog.redline.st/2010/12/12/redline-smalltalk-in-“small-talks”/

More than just Smalltalk is covered.

Rgs, James.

 

[Stefan Ram]

in my case, I would like (decent) C and C# support...

The memory model might be a problem with C. C allows to get
(and sometimes print) the addresses of objects and functions
and to allocate structures on the heap, to get the offset of
structure members and so on.

There is at least one Intel-PC-Emulator (386 based) for the
JVM: This will allow you to run 386 code which can come from
a C or C++ compiler on the JVM. An approach without such an
additional layer might not be possible. Possibly, you also
could install Microsoft® Windows and then run C# code.

However, it should be very difficult to access the real
environment in this case.

also maybe dynamic compilation, ...

We already have this in the JDK (but not in the JRE).

I am doing my own implementation of the JVM,

This should take a lot of time, but are you getting paid
for it? If not, it is really admirable if your are able to
manage your life in such a way that there even is enough
time for such projects!

 

[Arne Vajhøj]

in my case, I would like (decent) C and C# support...
also maybe dynamic compilation, ...

C# is done:

http://dev.mainsoft.com/Default.aspx?tabid=130#MS_IL_to_Java_ByteCode_Compilation

I am skeptical about whether C can run on the JVM as I suspect
some of the Java safeness features are implemented at the
JVM level.

There are plenty of dynamic compilation possibilities
for Java.

Arne

 

[BGB]

The memory model might be a problem with C. C allows to get
(and sometimes print) the addresses of objects and functions
and to allocate structures on the heap, to get the offset of
structure members and so on.

or add pointers, sizeof, offsetof, ..., to the VM...

Java doesn't need to use them, and possibly something like in .NET can
be done to deal with their existence:
untrusted code trying to use these features is immediately rejected.
nearly everything else then will pretend they don't exist.


I have already added struct support, and have considered some idle ideas
as to how to deal with unions (currently involving hacks with structs).

note: these pointers will be non-boxed "raw pointers".
currently, some notable restrictions are also placed on them (currently,
no pointer-based types support virtual methods, for possibly
obvious-enough reasons, ...).


the C compiler code may allow some extensions to ease working with
non-pointer reference types (and possibly extensions to support "opaque
strings"), hence some amount of code may be written which can gloss over
whether it is running in native land or tweaked-JVM land (I already do
this to some extent in my framework for other reasons).

hence, with pointer-free code it may be possible to produce non-extended
bytecode (and thus be usable on the standard JVM).

other extensions may also be used to ease interaction with the Java
object model, ...

so, IOW, something along vaguely similar lines to C++/CLI (although
starting with a C base, and adapting it to JVM-based technology).


admittedly, the above is a lot of why I am doing my own implementation:
there is probably little hope that Oracle would at all cooperate with
something like this (nor am I that inclined to wait for things to cycle
through their "process").


note, another reason for this:
it may help C code also gloss over whether it is running on a 32 or 64
bit target, which is currently an issue (I am imagining supporting
common binary code runnable on both Windows and Linux and in both 32 and
64 bit mode...).

There is at least one Intel-PC-Emulator (386 based) for the
JVM: This will allow you to run 386 code which can come from
a C or C++ compiler on the JVM. An approach without such an
additional layer might not be possible. Possibly, you also
could install Microsoft® Windows and then run C# code.

However, it should be very difficult to access the real
environment in this case.

yes...

this is exactly the sort of thing I don't want to do.
AFAIK, many prior "C to JVM" compilers have gone the route of simulating
byte addressable memory via arrays and similar, but this sucks too hard IMO.


direct access to the real environment is personally important (among
other things), and the awkwardness of this has actually been part of why
I have not used the JVM much in the past. (and my existing C codebase is
not exactly small...).

so, the C implementation needs to be all of memory, data, and ABI
compatible with native C code.

We already have this in the JDK (but not in the JRE).

yeah, because JDK includes javac and JRE does not?...

This should take a lot of time, but are you getting paid
for it? If not, it is really admirable if your are able to
manage your life in such a way that there even is enough
time for such projects!

well, I am a hobbyist...
but also a college student (currently going for a CS major), and
probably college students have a bit more free time than people with a
job...

it is being based on JavaME though, as there is little chance of me
writing an entire JavaSE implementation by myself (and I don't really
have much need to burn myself out trying to make AWT and Swing and so on
work...).

another issue is that there are some newer JVM features (generics, ...)
for which I don't have much up-to-date information regarding how they
are implemented (currently, I am not using them).


also, I have been fiddling around with compiler and VM technologies for
much of a decade, so I have a bit of a head start. for example, I ended
up more just writing a "JVM-like skin" for my existing stuff, rather
than having to start clean (implementing an entire VM framework from the
ground up).


similarly, I already have both a C compiler in place (for native code
though), as well as much of what is needed for a Java and C# compiler,
and for compiling to JBC (the main part needing work, not yet complete).
so, mostly the work is in bridging things and making it all work.


but, all of this is at-best "experimental"...

 

[BGB]

C# is done:

http://dev.mainsoft.com/Default.aspx?tabid=130#MS_IL_to_Java_ByteCode_Compilation

this is a way to do it...
admittedly it is a bit crufty IMO, but oh well.


I am doing a direct compiler (custom written, in C...).
I will see if I can make it workable within the confines of standard
bytecode (probably will prevent using 'unsafe' features though, but no
surprise there).

I am skeptical about whether C can run on the JVM as I suspect
some of the Java safeness features are implemented at the
JVM level.

yep.
partly it is VM-level safeness, and partly it is VM-level lack of unsafe
features (built in pointer and struct support, ...).

hence, VM extensions, ...

I don't want yet more dynamic languages, I want the ability to run C and
similar on the VM, hence part of why I am doing my own implementation
(and allowing for tweaking and extensions).

There are plenty of dynamic compilation possibilities
for Java.

agreed...

one can produce bytecode and shove it into the classloader, for example...

if one has JDK they can invoke javac (but, it is a little bit of a hack
though, and "compile this Java code and load it" is not exactly a
standard API feature, although yes, it is more standard than throwing
together ones' own VM to do the job).

....

but, alas...

 

[Arne Vajhøj]

this is a way to do it...
admittedly it is a bit crufty IMO, but oh well.

To support multiple .NET languages it may actually
be the smart way.

agreed...

one can produce bytecode and shove it into the classloader, for example...

if one has JDK they can invoke javac (but, it is a little bit of a hack
though, and "compile this Java code and load it" is not exactly a
standard API feature, although yes, it is more standard than throwing
together ones' own VM to do the job).

Java has had a documented compiler API since 1.6.

(I think that API is unnecessary convoluted, but it exists)

Arne

 

[Roedy Green]

in my case, I would like (decent) C and C# support...
also maybe dynamic compilation, ...

It would have to be a C-like language, or have lots of native code to
implement basic functionality. There are things you can do in C that
are forbidden in the JVM -- e.g. store data on top of other data
accessing it with a different descriptor, use pointers to change
arbitrary bits in ram.
--
Roedy Green Canadian Mind Products
http://mindprod.com

Doubling the size of a team will probably make it produce even more slowly.
The problem is the more team members, the more secrets, the less each team
member understands about how it all fits together and how his changes may
adversely affect others.

 

[BGB]

It would have to be a C-like language, or have lots of native code to
implement basic functionality. There are things you can do in C that
are forbidden in the JVM -- e.g. store data on top of other data
accessing it with a different descriptor, use pointers to change
arbitrary bits in ram.

actually, I was working on adding VM support for all of this (structs,
pointers, unions, ...), despite it being non-standard (so, C would work
on my VM, but, sadly, not on a standard JVM).

(as noted elsewhere, emulation is not a valid option in my case).


currently uncertain though is the 'best' way to handle C style linking
semantics (ideally without having to use VM extensions or redirect
function calls via API calls, IOW: call a method to call a function).

better would be to call directly via static method calls, but this is
complicated some by it being less obvious which functions exist in which
source modules (it would require a compiler trick to be able to figure
this out).


supporting non-class static methods or partial classes is a possible way
around this, but, sadly, this would also be a VM extension (if support
were added).

a non-class static method would simply be an 'invokestatic' with a
method where the class is null, which would be interpreted as being a
special case.


possible hack:
any class-names starting with '$' (or 'V_$' or similar) are not real
classes, but infact are considered as a part of the containing package.


also possible:
the in-classfile classname doesn't necessarily match the external
filename, and this may be used to support both partial and non-class
methods.

say:
myapp/Foo$14A9.class
myapp/Foo$3C9F.class
....

so, all would self-identify as "myapp/Foo", and all merge into a single
class in the VM.

myapp/_$2C9A.class
could identify itself simply as "myapp" and thus merge into the
containing package.

where after the '$' would be a hash or a gensym, so that clashes can be
avoided (if a hash, it could be a hash of the original filename or similar).


note:
a basic
"myapp/Foo.class" or "myapp/_.class" would not exist, as to help avoid
confusion with inner classes.


C scoping would work by assuming than an open-ended partial class exists
representing the entire C toplevel (any C functions would then be
assumed to be a method in this class).


I guess it is a big question over whether a standard JVM C-subset needs
to be able to exist, or if C support will simply not work on a standard
JVM...


or such...

 

[BGB]

To support multiple .NET languages it may actually
be the smart way.

well, this is assuming that one is supporting C# for sake of supporting
..NET, rather than supporting C# for sake of having "something like Java
but more C++ like and with more nifty features...".

if generalized (say, to support dynamically loading .NET assemblies),
this would mean essentially dragging around an implementation of both
VM's at the same time, and is not likely to be "that" much better than
the option of more simply using Mono and IKVM...


and, in my case, I could just go and finish writing the ECMA-335 loader
for my VM...

I decided against this though, as JBC seemed a little closer to my goals
(even though it would still require a little more hacking than MSIL
would have).


actually, less effort may well have just been to just use one of my own
bytecode formats, but I have my reasons.


like, options:
use JBC, potentially have to hack it all to hell to support wanted
featureset;
use MSIL, have wanted featureset, face architectural and usage-related
conflicts (likely more effort than hacking JBC);
use my own bytecode, but have no direct binary compatibility with
anything...

Java has had a documented compiler API since 1.6.

(I think that API is unnecessary convoluted, but it exists)

yes, ok.


may go look at it...