[ANN] Bootstrapping Ruby with MinGW: selfhosted and test sandbox!

[Luis Lavena]

Hello List,

I'm happy to announce that huge progress (hack-a-ton) was made this
weekend to bring a new implementation of One-Click Installer for
Windows!

This test version is based on the rake recipes I commented on previous
posts [1], and it download MinGW, MSYS and some Ruby dependencies to
get a working Ruby implementation using MinGW (i386-mingw32 platform).

I've packaged Ruby 1.8.6 (revision 15830) as 7zip package (2.7MB). You
can download it form the following URL:

http://dump.mmediasys.com/installer3/

The project and the recipes code can be downloaded from here:

http://code.mmediasys.com/installer3/latest.zip

I'll like to point a few things before everyone start using this ruby
version:

THIS BUILD OF RUBY IS *NOT* READY FOR PRODUCTION. USE IT ONLY FOR
TESTING PURPOSES.

All the enhancements got here will be injected back in the new version
of One-Click Installer for Windows.

Caveats:

1) Pre-built gems for i386-mswin32 are not compatible with this build
of ruby. You'll require to download sources of your favorite gems.

If you haven't created the sandbox environment using the recipes, you
can grab the baked mingw+msys sandbox from installer3 URL. You need to
put MinGW binaries and MSYS binaries in the PATH (in that order)
before firing the gem install or any build procedure.

2) Don't expect Rails, Mongrel, Merb, Sqlite3 or "name your favorite
gem" will work with this release out of the box (hey, this is not even
a release!).

3) The package contains the bare minimum stuff to serve as self
hosting environment:

Ruby 1.8.6
Zlib 1.2.3
OpenSSL 0.9.7c
Readline 5.2
Iconv alternative
RubyGems 1.0.1

Nothing more, nothing less.

4) Readline is still broken. I removed the readline tests from the
check procedure to continue working on this (was stuck for two months
without feedback). If you wanna help, search for my previous posts
about this issue.

For CI information:
1760 tests, 1343436 assertions, 2 failures, 0 errors

5) GNU Utils are a nightmare to get it working or find proper builds
for Windows. following _why advice [2], opted for win_iconv which
seems working (but there aren't tests for it).

6) Microsoft Installer (MSI) recipes are still missing. Future
releases will be use WiX tools and will provide patches between
releases.

7) Contributions are highly welcome! I'm using Bazaar as VCS since
Windows support for Git (the new cool tool) is still behind...

Remember: everything shown here will became part of the next One-Click
Installer. As you can see, I'm trying to keep it to the minimum. Later
we can discuss "addons" or different versions that are target
different usage scenarios.

Check my blog [2] for upcoming hack-a-ton dates and keep an eye on
#ruby-lang if you wanna help ;-)

Regards and everybody have a nice week!

[1] http://groups.google.com/group/ruby-talk-google/browse_thread/thread/1e2f1b9a3d611e3b/
[2] http://blog.mmediasys.com
--
Luis Lavena
Multimedia systems
-
Human beings, who are almost unique in having the ability to learn
from
the experience of others, are also remarkable for their apparent
disinclination to do so.
Douglas Adams

 

[Sean O'Halpin]

Hello List,

I'm happy to announce that huge progress (hack-a-ton) was made this
weekend to bring a new implementation of One-Click Installer for
Windows!

Hi Luis.

May I congratulate you on your perseverance! I tried something similar
a couple of years ago and gave up in frustration (and ditched Windows
in the process This is a great service to the Ruby on Windows
community. Well done.

Regards,
Sean

 

[rogerdpack]

Note that binary gems don't work only because they install their
binary aspect into the wrong directory (and, if you have the newer
versions of rubygems, auto-install a version which doesn't exist so
you'd need to specify you want the win32 gem). Once you install it
you can easily copy the .so's to the right directory and they will
'probably' work just fine.

Note that mongrel works if you do this then just go into mongrel.rb
and change their 'win32' regexes to be win32|mingw

Good luck!
-R

 

[Luis Lavena]

Note that binary gems don't work only because they install their
binary aspect into the wrong directory (and, if you have the newer
versions of rubygems, auto-install a version which doesn't exist so
you'd need to specify you want the win32 gem). Once you install it
you can easily copy the .so's to the right directory and they will
'probably' work just fine.

Not that, RubyGems consider each platform by it's own. I snot the same
darwin8 to cygwin, so mingw32 shouldn't be considered a compatible
match to mswin32.

In a overlooked way you could be correct, but that is too simplistic
way to look things. When you start seeing errors due exception thrown
inside shared object we can find several and big issues.

Consider that the code generated by VC6 is not the same MinGW
generates. It even differs in several places on how it do things.

Note that mongrel works if you do this then just go into mongrel.rb
and change their 'win32' regexes to be win32|mingw

The gems mongrel provides right now are tested for mswin32. Now that
this build of mingw32 seems a good candidate for replacement, I'll try
to fix as much as possible the code related to this on the project I
contribute back.

But again, without overlooking at the important issues behind this :-D

Good luck!

Thank you Roger, I'll really need it ;-)

Regards,

 

[Nobuyoshi Nakada]

Hi,

At Tue, 25 Mar 2008 06:04:52 +0900,
Luis Lavena wrote in [ruby-talk:295564]:

Not that, RubyGems consider each platform by it's own. I snot the same
darwin8 to cygwin, so mingw32 shouldn't be considered a compatible
match to mswin32.

Darwin and cygwin are completely different systems, but mingw32
and mswin32 are for the same system.

Not only mingw, gcc is basically intended to be binary
interface compatible with the system native development
environments. So cygwin is the special case in this sense.

In a overlooked way you could be correct, but that is too simplistic
way to look things. When you start seeing errors due exception thrown
inside shared object we can find several and big issues.

Consider that the code generated by VC6 is not the same MinGW
generates. It even differs in several places on how it do things.

Binary interface compatible means that they use same calling
convension, same utility functions, same runtime DLL etc, but
not generated code are identical.

 

[Luis Lavena]

Hey Nobu,

Hi,

At Tue, 25 Mar 2008 06:04:52 +0900,
Luis Lavena wrote in [ruby-talk:295564]:

Not that, RubyGems consider each platform by it's own. I snot the same
darwin8 to cygwin, so mingw32 shouldn't be considered a compatible
match to mswin32.

Darwin and cygwin are completely different systems, but mingw32
and mswin32 are for the same system.

Not only mingw, gcc is basically intended to be binary
interface compatible with the system native development
environments. So cygwin is the special case in this sense.

I just tried to be dramatic, also exposing that RubyGems will not
consider mingw a compatible platform to mswin32, since it's using
RbConfig::CONFIG['arch'] to determine it's platform.

I'll like to point htat rubygems 'darwin' is compatible with 'darwin7'
and 'darwin8', but version 7 is not compatible with 8, like happens
with buils with VC6 and VC8/9.

Binary interface compatible means that they use same calling
convension, same utility functions, same runtime DLL etc, but
not generated code are identical.

You're are correct, but real life prove me wrong.

Even that MinGW generates msvcrt-ruby18.dll library, I cannot put a
pre-compiled extension build with VC6 to run over the MinGW build, it
just crash.

(Tried several times during the testing procedure).

Everything depends on the complexity of things the extensions perform,
but instead of guessing and "hoping" not to crash, I put my statement
on bold, underlined text: extensions are not easily interchangeable
between these two build systems.

To solve this I'll require a lot of deep tracing inside ruby, and I
still want to fix the redirected IO issues I had with Readline and
still noone replied.

Thank you for your reply.

Regards,

 

[James Tucker]

Hey Nobu,

Hi,

At Tue, 25 Mar 2008 06:04:52 +0900,
Luis Lavena wrote in [ruby-talk:295564]:

Note that binary gems don't work only because they install their
binary aspect into the wrong directory (and, if you have the newer
versions of rubygems, auto-install a version which doesn't exist so
you'd need to specify you want the win32 gem). Once you install it
you can easily copy the .so's to the right directory and they will
'probably' work just fine.

Not that, RubyGems consider each platform by it's own. I snot the
same
darwin8 to cygwin, so mingw32 shouldn't be considered a compatible
match to mswin32.

Darwin and cygwin are completely different systems, but mingw32
and mswin32 are for the same system.

Not only mingw, gcc is basically intended to be binary
interface compatible with the system native development
environments. So cygwin is the special case in this sense.

I just tried to be dramatic, also exposing that RubyGems will not
consider mingw a compatible platform to mswin32, since it's using
RbConfig::CONFIG['arch'] to determine it's platform.

I'll like to point htat rubygems 'darwin' is compatible with 'darwin7'
and 'darwin8', but version 7 is not compatible with 8, like happens
with buils with VC6 and VC8/9.

Binary interface compatible means that they use same calling
convension, same utility functions, same runtime DLL etc, but
not generated code are identical.

You're are correct, but real life prove me wrong.

Even that MinGW generates msvcrt-ruby18.dll library, I cannot put a
pre-compiled extension build with VC6 to run over the MinGW build, it
just crash.

(Tried several times during the testing procedure).

Everything depends on the complexity of things the extensions perform,
but instead of guessing and "hoping" not to crash, I put my statement
on bold, underlined text: extensions are not easily interchangeable
between these two build systems.

To solve this I'll require a lot of deep tracing inside ruby, and I
still want to fix the redirected IO issues I had with Readline and
still noone replied.

As you know Luis, I'm desperately trying to make time to come and help
you, sort out some of the code i can donate, etc. Sadly, time is
lacking. Anyway, it seems this kind of project would be ideal for the
Ruby Mendicant project. I know there are a lot of the long-term
rubyists who simply say "Oh we don't care about Windows" or "It's a
broken platform" / whatever. The fact is, actually getting a
performant stable interpreter on the platform would probably do more
for the language than any other single action. Of course OCI download
numbers from rubyforge can easily attest to this.

 

[Gregory Brown]

Hey Nobu,

Hi,
At Tue, 25 Mar 2008 06:04:52 +0900,
Luis Lavena wrote in [ruby-talk:295564]:
Note that binary gems don't work only because they install their
binary aspect into the wrong directory (and, if you have the newer
versions of rubygems, auto-install a version which doesn't exist so
you'd need to specify you want the win32 gem). Once you install it
you can easily copy the .so's to the right directory and they will
'probably' work just fine.
Not that, RubyGems consider each platform by it's own. I snot the
same
darwin8 to cygwin, so mingw32 shouldn't be considered a compatible
match to mswin32.
Darwin and cygwin are completely different systems, but mingw32
and mswin32 are for the same system.
Not onlymingw, gcc is basically intended to be binary
interface compatible with the system native development
environments. So cygwin is the special case in this sense.

I just tried to be dramatic, also exposing that RubyGems will not
considermingwa compatible platform to mswin32, since it's using
RbConfig::CONFIG['arch'] to determine it's platform.

I'll like to point htat rubygems 'darwin' is compatible with 'darwin7'
and 'darwin8', but version 7 is not compatible with 8, like happens
with buils with VC6 and VC8/9.

You're are correct, but real life prove me wrong.

Even thatMinGWgenerates msvcrt-ruby18.dll library, I cannot put a
pre-compiled extension build with VC6 to run over theMinGWbuild, it
just crash.

(Tried several times during the testing procedure).

Everything depends on the complexity of things the extensions perform,
but instead of guessing and "hoping" not to crash, I put my statement
on bold, underlined text: extensions are not easily interchangeable
between these two build systems.

To solve this I'll require a lot of deep tracing inside ruby, and I
still want to fix the redirected IO issues I had with Readline and
still noone replied.

As you know Luis, I'm desperately trying to make time to come and help
you, sort out some of the code i can donate, etc. Sadly, time is
lacking. Anyway, it seems this kind of project would be ideal for the
Ruby Mendicant project.

It's an interesting idea, but my project is not Google's Summer of
Code (which I think this idea is better suited for).

Unfortunately, M$ support a very bad project for me, because I don't
even own a machine with Windows on it, and have moved all my
commercial work off of the platform too. The only tie I have left to
the Microsoft world I slide underneath via ODBC....

For RubyMendicant, since it isn't really the same as the GSoC (ie, I'm
taking off from my commercial work to do this!),
I really need to pick projects that I have expertise in along with a
strong interest so that people's donations are used most effectively.

Windows unfortunately doesn't fit that bill, even if the project would
be well worth it for someone better suited.

I have selected 3 projects from my ideas list, and am mostly limiting
the potential projects to these for now:
http://rubymendicant.wikidot.com/projects

Though I won't ignore suggested project ideas, I'm asking folks to
rally behind one or more of those projects, since I've already seen
some support for them and have a plan in mind... it's starting to get
too close to the start date to consider entirely new ideas.

I know there are a lot of the long-term
rubyists who simply say "Oh we don't care about Windows" or "It's a
broken platform" / whatever. The fact is, actually getting a
performant stable interpreter on the platform would probably do more
for the language than any other single action. Of course OCI download
numbers from rubyforge can easily attest to this.

I agree this is important. Maybe suggest this as a GSoC project idea.

http://rubycentral.com/projects/gsoc-2008/ideas-for-gsoc-2008

-greg

 

[James Tucker]

It's an interesting idea, but my project is not Google's Summer of
Code (which I think this idea is better suited for).

Unfortunately, M$ support a very bad project for me, because I don't
even own a machine with Windows on it, and have moved all my
commercial work off of the platform too. The only tie I have left to
the Microsoft world I slide underneath via ODBC....

For RubyMendicant, since it isn't really the same as the GSoC (ie, I'm
taking off from my commercial work to do this!),
I really need to pick projects that I have expertise in along with a
strong interest so that people's donations are used most effectively.

Windows unfortunately doesn't fit that bill, even if the project would
be well worth it for someone better suited.

I have selected 3 projects from my ideas list, and am mostly limiting
the potential projects to these for now:
http://rubymendicant.wikidot.com/projects

Absolutely fair enough

semi-P.S. Good luck with project.

 

[Luis Lavena]

As you know Luis, I'm desperately trying to make time to come and help
you, sort out some of the code i can donate, etc. Sadly, time is
lacking.

I know about you James, and I'm not pointing fingers to anyone, but I
heard lot of folks scream that OCI have problems, but they, even have
plenty of C or Ruby knowledge, don't offer their help.

Anyway, it seems this kind of project would be ideal for the
Ruby Mendicant project.

from what I have seen the Mendicant project is aimed to other kind of
project and not GSoC or targeting Windows.

The only thing I'll like to add is: whatever code you write, please
try be the most cross-platform aware you can :-D

I know there are a lot of the long-term
rubyists who simply say "Oh we don't care about Windows" or "It's a
broken platform" / whatever. The fact is, actually getting a
performant stable interpreter on the platform would probably do more
for the language than any other single action. Of course OCI download
numbers from rubyforge can easily attest to this.

Yes, it seems that today, noone ever used or started computing running
over a M$ OS... DOS or even Windows.

All them reject their roots ;-)

I'll keep Windows as my main platform, since I'm quite comfortable
with it. To be honest the only language that give me so many troubles
with it is Ruby (and I can provide a long list of languages and tools
I have used since I started with computers late in the 80's).

Regards,

 

[M. Edward (Ed) Borasky]

I'll keep Windows as my main platform, since I'm quite comfortable
with it. To be honest the only language that give me so many troubles
with it is Ruby (and I can provide a long list of languages and tools
I have used since I started with computers late in the 80's).

Long ago in a galaxy far away, there were *major* differences between
the ActiveState version of Perl and other versions, even on Windows. In
the end, someone sat on ActiveState about that and it got fixed. I think
the same thing needs to happen to Ruby 1.8.x -- Ruby 1.8.x should be
Ruby 1.8.x regardless of whether it's Cygwin, MSYS, DOS, Windown
95/98/ME, Windows 2000/XP/Vista, etc.

 

[James Tucker]

I know about you James, and I'm not pointing fingers to anyone, but I
heard lot of folks scream that OCI have problems, but they, even have
plenty of C or Ruby knowledge, don't offer their help.

Absolutely, I understand. Among other things I'm still setting up my
build management servers and so on for tracking external projects like
OCI / rubygems / other dependencies.

Having places to go and experiment helps a lot, and also learning to
deal with niggly problems, like sorting out relative vs. full path
issues and so on (errors in both ruby's makefiles for openssl, and in
rubygems test suite, for one example).

I think you've been through some of these fixes already, and I know
Eric was looking for you last night regarding some newer stuff with
rubygems, but keeping up with everything, whilst running a startup, is
kinda crazy. Basically, once I have my build setups stable (if I ever
get enough time to do that...), I'll be publishing them so that we can
all start working off a similar base, and problems should get more
extensively replicable.

from what I have seen the Mendicant project is aimed to other kind of
project and not GSoC or targeting Windows.

The only thing I'll like to add is: whatever code you write, please
try be the most cross-platform aware you can :-D

I think the only major 'ruby platform' I don't test on is Cygwin.
Whether or not to count that as a *nix or a windows style OS is
extremely foggy, as it depends what you're doing (paths vs A[PB]I, etc
etc).

Yes, it seems that today, noone ever used or started computing running
over a M$ OS... DOS or even Windows.

All them reject their roots ;-)

I still love my Amiga, it's just a 68030 doesn't cut it these days ;-)

I'll keep Windows as my main platform, since I'm quite comfortable
with it. To be honest the only language that give me so many troubles
with it is Ruby (and I can provide a long list of languages and tools
I have used since I started with computers late in the 80's).

I 'grew up' consulting in the windows world, although I'll relatively
happily use most things. I'm still hopeful that one day I'll find an
OS I actually like.

I gotta say, the core interpreter compiles pretty easily on most
platforms. The ext/ dir and dependencies however...

 

[Michal Suchanek]

I know about you James, and I'm not pointing fingers to anyone, but I
heard lot of folks scream that OCI have problems, but they, even have
plenty of C or Ruby knowledge, don't offer their help.

For me either OCI or the new installer is good enough. I know there
are problems with the way OCI is built but they did not affect me so
far

I see that you have put lots of time into researching how things are
done on Windows, many thanks for that. I do not have much experience
with Windows programming but I hope that now applying fixes and
enhancements to Ruby on Windows will be as easy and streamlined as on
other platforms.

As far as I know for the few scripts I use on Windows I could replace
the OCI with the mingw built ruby.

Yes, it seems that today, noone ever used or started computing running
over a M$ OS... DOS or even Windows.

All them reject their roots ;-)

Actually I started with the ZX Spectrum BASIC IDE(if you may call it
one). Of course, DOS was better than that, and Windows was eventually
better than DOS but before that I found GNU/Linux is even better in
not getting into my way. When I went back to Windows I found that they
are quite hard to work with.
This is changing somewhat with the possibility to use alternate GUI
shell, MSYS, and whatnot but it's still a hassle to set up, and stuff
breaks if you do not use the default settings for everything. I do not
reject Windows, and I still use dosemu but once better alternative
than Linux and BSD appears I will leave them behind as well.

Hate all the OSes equally for their inflexibility and poor design ;-)

I'll keep Windows as my main platform, since I'm quite comfortable
with it. To be honest the only language that give me so many troubles
with it is Ruby (and I can provide a long list of languages and tools
I have used since I started with computers late in the 80's).

Yes, each pick their poison

Thanks

Michal

 

[rogerdpack]

I think very useful would be a mingw build of ruby that included some
popular gems pre-built (that is...<cough> mongrel, sqlite, mysql,
maybe ruby-debug). Assuming that those gems only need to be compiled
in mingw and it works like a charm.

Then people could use it as their dev environment and 'test it
out' (if they work in rails, that is), and also give feedback, and
avoid the pain of compiling their own extensions (as easy as that is
now, thanks to Luis' build helpers).
(I assume you also tried http://gnuwin32.sourceforge.net/packages/readline.htm
this readline?)

GL!
-R

 

[ara.t.howard]

I think very useful would be a mingw build of ruby that included some
popular gems pre-built (that is...<cough> mongrel, sqlite, mysql,
maybe ruby-debug). Assuming that those gems only need to be compiled
in mingw and it works like a charm.

but, if you have ruby and mingw installed all you need to do to
install these is 'gem install sqlite', etc. i compiled the gsl this
way and it was a peice of cake. ruby + compiler = joy.

-a

 

[Luis Lavena]

I think very useful would be a mingw build of ruby that included some
popular gems pre-built (that is...<cough> mongrel, sqlite, mysql,
maybe ruby-debug). Assuming that those gems only need to be compiled
in mingw and it works like a charm.

Well, part of the idea of getting ruby build with Ruby is put it on CI
and forget about it.
Also, add it as platform to the CI and build some of the projects you
mention, with the idea of generate as output the pre-compiled
packages

Then people could use it as their dev environment and 'test it
out' (if they work in rails, that is), and also give feedback, and
avoid the pain of compiling their own extensions (as easy as that is
now, thanks to Luis' build helpers).
(I assume you also tried http://gnuwin32.sourceforge.net/packages/readline.htm
this readline?)

Yeah, tried it, all the flavors or Readline. from 4.3 source to 5.2
using mingwPORTs.

GL!

 

[Luis Lavena]

but, if you have ruby and mingw installed all you need to do to
install these is 'gem install sqlite', etc. i compiled the gsl this
way and it was a peice of cake. ruby + compiler = joy.

That's the idea! Since I'll not be able to build every gem for ruby
out there, make it easy or put a familiar environment that users than
run for themselves.

That also requires that the gems you're talking about consider Windows
(VC or MinGW or whatever) as something valid to get it up and running
with it.

Regards,

 

[Joel VanderWerf]

This is all working very smoothly for me, following the recipe.

I ended up with a ruby/mingw32 installation which my program can use to
dynamically compile its parts that are generated C code, and run 20%-50%
faster than using msvc6 and oci. This is also far preferable than
requiring users to have msvc6 installed if they make changes that
require recompilation of the generated C code.

The only slight hiccup was figuring out that I needed to put
sandbox/ruby_mingw/bin on my PATH, but that's really pretty obvious.

I hope someone is able to fix readline, though...

Thanks, Luis!

 

[Luis Lavena]

This is all working very smoothly for me, following the recipe.

I ended up with a ruby/mingw32 installation which my program can use to
dynamically compile its parts that are generated C code, and run 20%-50%
faster than using msvc6 and oci. This is also far preferable than
requiring users to have msvc6 installed if they make changes that
require recompilation of the generated C code.

Great, good to know

The only slight hiccup was figuring out that I needed to put
sandbox/ruby_mingw/bin on my PATH, but that's really pretty obvious.

Well, the idea of sandbox/ruby_mingw is be there until it get properly
packaged ;-)

I suggest you put it into C:/Ruby or something like that, or have a
inside your %HOME% a full stack of ruby versions like I do:

ruby-185-p114-VC6-rubygems-094
ruby-186-p111-VC6-rubygems-094
ruby-186-p111-VC6-rubygems-094-oci
ruby-186-p111-VC6-rubygems-095
ruby-186-p111-VC6-rubygems-101
ruby-186-svn-MINGW-rubygems-101

I hope someone is able to fix readline, though...

I want to try the non-blocking patch Park Heesob created for IO.gets,
I don't think it's related, but smell fishy...

Thanks, Luis!

No problem man, I hope collect a list of the most used gems that want
get proper support for MinGW build, now that RubyGems is fixed ;-)

Regards,

 

[Michal Suchanek]

This is all working very smoothly for me, following the recipe.

I ended up with a ruby/mingw32 installation which my program can use to
dynamically compile its parts that are generated C code, and run 20%-50%
faster than using msvc6 and oci. This is also far preferable than
requiring users to have msvc6 installed if they make changes that
require recompilation of the generated C code.

The only slight hiccup was figuring out that I needed to put
sandbox/ruby_mingw/bin on my PATH, but that's really pretty obvious.

I hope someone is able to fix readline, though...

I have read something about pure ruby readline on this list. There's
something on RAA but the linked site is down.

Thanks

Michal