Get your hands dirty: Help bootstrap Ruby on Windows.

[Luis Lavena]

This was asked, and asked, and asked too many times, so here we go.

Since I don't want to hold anyone to try it, I'll like to share some
progress I've made in this area.
I'll like to make a few points clear before:

1) VC6 vs. VC8 vs. MinGW have no room in this discussion. I took the
MinGW path since was the only one that could be bootstrapped from
scratch.

2) This is a work in progress. Even that I'm 110% busy with work,
interest on this seems it deserve be shared, no mater how "alpha" the
whole process is.

3) Don't blame me if isn't work or your Rails application don't work
either. Isn't complete!. Instead: grab the code, hack it and send me
your patches.

4) Make code talk for you. I've heard this too many times on other
projects, so please move from simple comments to share your ruby-fu.

5) Part of this project is bring a new build procedure to One-Click
Installer, if we succeed, everybody will enjoy the benefits.

For those who aren't aware, I'm the new maintainer of OCI (One-Click
Ruby Installer for Windows). Curt give me the kingdom keys a few
months back and the latest release was completely build in my office
computer.

OCI, in its current state is a 'Package of Ruby Tools and libraries',
not a Installer. There are plans to change that.

So, now the code:

http://code.mmediasys.com/installer3/latest.zip

This is a pure-ruby solution that download and setup a mingw+msys
environment to get Ruby build.

It uses some rake recipes to download all the needed packages
At this time Ruby 1.8.6 p111 is targeted, but targeting Ruby 1.9 will
no have mayor issues.

Dependencies are not compiled, just downloaded for the time being.
Tested all the scenarios (pre-built and compile from source) and there
are issues with them which I reported previously.

To bootstrap and use this project, you need:
- Ruby 1.8.5 or greater
- Rake 0.7.3 or greater
- Tar [1] and Unzip [2] binary packages from GnuWin32

If someone can provide me a way to unpack 4MB zip, tgz and tar.bz2
files without external dependencies (just ruby) -- you're welcome.

The rake tasks available:
rake check # Run tests for the interpreter in the sandbox.
rake clean # Remove any temporary products.
rake clobber # Remove any generated file.
rake compile # Compile the interpreter.
rake configure # Run the configure process for the interpreter.
rake download # Download all components.
rake extract # Extract all downloaded components.
rake install # Install the interpreter in the sandbox.
rake prepare # Prepare the freshly extracted components.

clean will remove the sandbox and clobber will remove all the
downloaded files.

The step sequence is:

download, extract, prepare, configure, compile, install, check

you will find the installed ruby in sandbox/ruby_mingw

You need to install before check, so the check process use the sandbox
interpreter instead of the one you're using to build it.

check will fail in the readline tests, since I cannot get them working
properly.

I suggest to those who are interested, help me get this working.

====

Checkout code:

I'm using Bazaar (bzr) as distributed version control software:

http://www.bazaar-vcs.org/

bzr branch http://code.mmediasys.com/installer3/dev my-local-branch

====

[1] This ships inside libarchive-2.4.0 named as 'bsdtar.exe'
http://gnuwin32.sourceforge.net/packages/libarchive.htm

[2] http://gnuwin32.sourceforge.net/packages/unzip.htm

--
Luis Lavena
Multimedia systems
-
A common mistake that people make when trying to design
something completely foolproof is to underestimate
the ingenuity of complete fools.
Douglas Adams

 

[Bill Kelly]

1) VC6 vs. VC8 vs. MinGW have no room in this discussion. I took the
MinGW path since was the only one that could be bootstrapped from
scratch.

Just to clarify... What does "bootstrapped from scratch" mean
in this context?


Regards,

Bill

 

[James Tucker]

Just to clarify... What does "bootstrapped from scratch" mean
in this context?

As low as he (we) can go. At the moment, it's in ruby (as per the code
presented and requirements), but some of my ideas had most of the
stack from a .cmd file - although that may be excessive.

js or .vb via WSH anyone?

 

[Luis Lavena]

From: "Luis Lavena" <[email protected]>




Just to clarify... What does "bootstrapped from scratch" mean
in this context?

Don't know if that is the right terminology (since I'm not a native
english speaker) but this means:

- Grab MinGW+MSYS
- Compile Ruby
- Package it
- Repeat!

Be usable so you can repeat the whole process with the new version of
the interpreter you just compiled

So future version use the new one, is like what Rubinius is doing:
bootstrapping itself using a previous ruby installation.

Hope this clarify things a bit.

 

[Luis Lavena]

As low as he (we) can go. At the moment, it's in ruby (as per the code
presented and requirements), but some of my ideas had most of the
stack from a .cmd file - although that may be excessive.

js or .vb via WSH anyone?

The idea is Bootstrap Ruby with Ruby, like Rubinius guys are doing.

I must admit that have extensive batch files that I use to bootstrap
ruby with VC6 at office, but maintain .CMD and workaround its
limitations don't worth the effort.

Also, we are using Ruby for the fun of it, right? ;-)

 

[James Tucker]

The idea is Bootstrap Ruby with Ruby, like Rubinius guys are doing.

I must admit that have extensive batch files that I use to bootstrap
ruby with VC6 at office, but maintain .CMD and workaround its
limitations don't worth the effort.

Also, we are using Ruby for the fun of it, right? ;-)

Right, shame one can't do the whole implement Jump, branch and then
move on up thing though... ;-)

 

[Nobuyoshi Nakada]

Hi,

At Fri, 4 Jan 2008 05:29:58 +0900,
Luis Lavena wrote in [ruby-talk:285910]:

So, now the code:

http://code.mmediasys.com/installer3/latest.zip

This is a pure-ruby solution that download and setup a mingw+msys
environment to get Ruby build.

Seems manifest task in ruby18.rake doesn't close the target
file, so the file may be incomplete at the end of the task.

 

[Luis Lavena]

Hi,

At Fri, 4 Jan 2008 05:29:58 +0900,
Luis Lavena wrote in [ruby-talk:285910]:

So, now the code:

This is a pure-ruby solution that download and setup a mingw+msys
environment to get Ruby build.

Seems manifest task in ruby18.rake doesn't close the target
file, so the file may be incomplete at the end of the task.

Thank you Nobu, I didn't give attention to manifest since I was using
WiX Heat to collect all the files.

Change committed and pushed updated branch and zip file.

Regards,

 

[M. Edward (Ed) Borasky]

Luis Lavena wrote:
[snip]

This is a pure-ruby solution that download and setup a mingw+msys
environment to get Ruby build.

It uses some rake recipes to download all the needed packages
At this time Ruby 1.8.6 p111 is targeted, but targeting Ruby 1.9 will
no have mayor issues.

Dependencies are not compiled, just downloaded for the time being.
Tested all the scenarios (pre-built and compile from source) and there
are issues with them which I reported previously.

To bootstrap and use this project, you need:
- Ruby 1.8.5 or greater
- Rake 0.7.3 or greater
- Tar [1] and Unzip [2] binary packages from GnuWin32

Aren't they in MSys?

[snip]

I suggest to those who are interested, help me get this working.

This does look like fun. I have a spare Windows XP license I'm not using
and a Linux hosted VMware Workstation 6. Are the compiler, linker,
"tar" and zip/gzip/bzip2 the only external dependencies you have at the
moment?

 

[Luis Lavena]

Luis Lavena wrote:

[snip]

To bootstrap and use this project, you need:
- Ruby 1.8.5 or greater
- Rake 0.7.3 or greater
- Tar [1] and Unzip [2] binary packages from GnuWin32

Aren't they in MSys?

Chicken-egg situation:

They are inside the compressed tar.gz and tar.bz2 files... how I'm
supposed to get them if I don't decompress them first? :-D

So: if someone have a pure-ruby (with aditional gem or extension) that
allow us extract content from:

- .zip (deflate) files
- .gz (of .tar.gz) files
- .bz2 (of .tar.bz2) files

All without these dependencies, feel free to modify rake/
extracttask

This does look like fun. I have a spare Windows XP license I'm not using
and a Linux hosted VMware Workstation 6. Are the compiler, linker,
"tar" and zip/gzip/bzip2 the only external dependencies you have at the
moment?

You don't need to manually download MinGW or MSYS, only you need get
your hands at tar and unzip.
(also, have a working ruby, which you could use OCI for that) ;-)

The compiler, linker and dependencies are downloaded automatically.

Right now only two "dependencies" are in place:

- Zlib 1.2.3
- Readline 5.0

I stopped at readline since I got some serious issues with
Readline.readline, as posted before to this list and ruby-core.

The thing is that those worked in the past, but right know they don't.

Each time I add a dependency (to get the bundled extension built), I
do a 'test-all' procedure and see if it's passing their self-tests.

Regards,

 

[Michal Suchanek]

Luis Lavena wrote:

[snip]

To bootstrap and use this project, you need:
- Ruby 1.8.5 or greater
- Rake 0.7.3 or greater
- Tar [1] and Unzip [2] binary packages from GnuWin32

Aren't they in MSys?

Chicken-egg situation:

They are inside the compressed tar.gz and tar.bz2 files... how I'm
supposed to get them if I don't decompress them first? :-D

So: if someone have a pure-ruby (with aditional gem or extension) that
allow us extract content from:

- .zip (deflate) files
- .gz (of .tar.gz) files
- .bz2 (of .tar.bz2) files

All without these dependencies, feel free to modify rake/
extracttask

If youcan afford relying on ruby stdlib, there is zlib which can do
deflate and gunzip.

There is minitar hosted on rubyforge under ruwiki (does not work on
unseekable files like stdin but works on gzipped files by using zlib).
There is rubyzip (untested, uses zlib for compression). bzip2 remains
to be solved but hopefully the basic stuff can be obtained as gzipped
archives.

http://rubyforge.org/frs/?group_id=84
http://rubyforge.org/frs/?group_id=909

Thanks

Michal

 

[Michal Suchanek]

Luis Lavena wrote:

[snip]

To bootstrap and use this project, you need:
- Ruby 1.8.5 or greater
- Rake 0.7.3 or greater
- Tar [1] and Unzip [2] binary packages from GnuWin32

Aren't they in MSys?

Chicken-egg situation:

They are inside the compressed tar.gz and tar.bz2 files... how I'm
supposed to get them if I don't decompress them first? :-D

So: if someone have a pure-ruby (with aditional gem or extension) that
allow us extract content from:

- .zip (deflate) files
- .gz (of .tar.gz) files
- .bz2 (of .tar.bz2) files

All without these dependencies, feel free to modify rake/
extracttask

If youcan afford relying on ruby stdlib, there is zlib which can do
deflate and gunzip.

There is minitar hosted on rubyforge under ruwiki (does not work on
unseekable files like stdin but works on gzipped files by using zlib).
There is rubyzip (untested, uses zlib for compression). bzip2 remains
to be solved but hopefully the basic stuff can be obtained as gzipped
archives.

http://rubyforge.org/frs/?group_id=84
http://rubyforge.org/frs/?group_id=909

And you can get bzip2 at
http://www.bzip.org/1.0.4/bzip2-104-x86-win32.exe - plain win32
executable.

Thanks

Michal

 

[Luis Lavena]

If youcan afford relying on ruby stdlib, there is zlib which can do
deflate and gunzip.

There is minitar hosted on rubyforge under ruwiki (does not work on
unseekable files like stdin but works on gzipped files by using zlib).
There is rubyzip (untested, uses zlib for compression). bzip2 remains
to be solved but hopefully the basic stuff can be obtained as gzipped
archives.

http://rubyforge.org/frs/?group_id=84
http://rubyforge.org/frs/?group_id=909

Thank you Michal,

Since I was moving from a batch-build process to a ruby/rake build
process, a few external tools remains as requirement.

MinGW packages are distributed in tar.gz format [1] (which I guess map
perfect for Minitar project), but MSYS provides tar.bz2 files [2],
which will still need the external tool to uncompress.

In this case, tar.bz2 files can be uncompressed with bsdtar, which is
the best implementation I've tested that work with all the file
formats. (tgz, tar.gz and tar.bz2)

There is also the memory usage and speed situation, extract 8MB files
can take a bit slow using a pure-ruby implementation.

[1] http://sourceforge.net/project/showfiles.php?group_id=2435&package_id=82723&release_id=237094
[2] http://sourceforge.net/project/showfiles.php?group_id=2435&package_id=24963&release_id=46827

Regards,

 

[Luis Lavena]

And you can get bzip2 athttp://www.bzip.org/1.0.4/bzip2-104-x86-win32.exe- plain win32
executable.

Thank you again.

I was using bsdtar from libarchive 2.4.0 mostly because it offered a
one-step unpack procedure:

tar xzf
(.tar, tar.gz and .tgz files)

tar jxf
(.tar.bz2)

Which is similar to the commands executed under *nix.

Anyhow this could be changed, patchs are welcome

Regards,

 

[Austin Ziegler]

They are inside the compressed tar.gz and tar.bz2 files... how I'm
supposed to get them if I don't decompress them first? :-D

So: if someone have a pure-ruby (with aditional gem or extension) that
allow us extract content from:

- .zip (deflate) files
- .gz (of .tar.gz) files
- .bz2 (of .tar.bz2) files

All without these dependencies, feel free to modify rake/
extracttask

gz and .tar.gz files aren't hard to deal with since a working Ruby
1.8.5 is a prerequisite. Look at minitar, and the zlib library is part
of a working Ruby 1.8.5 install.

-austin

 

[Michal Suchanek]

If youcan afford relying on ruby stdlib, there is zlib which can do
deflate and gunzip.

There is minitar hosted on rubyforge under ruwiki (does not work on
unseekable files like stdin but works on gzipped files by using zlib).
There is rubyzip (untested, uses zlib for compression). bzip2 remains
to be solved but hopefully the basic stuff can be obtained as gzipped
archives.

http://rubyforge.org/frs/?group_id=84
http://rubyforge.org/frs/?group_id=909

Thank you Michal,

Since I was moving from a batch-build process to a ruby/rake build
process, a few external tools remains as requirement.

MinGW packages are distributed in tar.gz format [1] (which I guess map
perfect for Minitar project), but MSYS provides tar.bz2 files [2],
which will still need the external tool to uncompress.

In this case, tar.bz2 files can be uncompressed with bsdtar, which is
the best implementation I've tested that work with all the file
formats. (tgz, tar.gz and tar.bz2)

There is also the memory usage and speed situation, extract 8MB files
can take a bit slow using a pure-ruby implementation.

The decompression is handled by zlib so this should not be that much
of an issue. Still people extracting on slow machines could see the
difference, and there should be no problem with including these unpack
tools in the archive containing the build scripts.
On the other hand, the minitar might be useful if you wanted to do
something special with symlinks (currently it does not extract them at
all). Maintaining a patched minitar might be easier than maintaining a
patched bsdtar binary.

The rubyzip implementation does not include a working command line
tool as far as I can tell. A poor man's unzip using the library is
simple but I am not sure it is of much use. Perhaps an external unzip
tool would be easier.

Thanks

Michal

#unzip.rb
# The example:
#Zip::ZipInputStream:pen("my.zip") {
# |io|
#
# while (entry = io.get_next_entry)
# puts "Contents of #{entry.name}: '#{io.read}'"
# end
# }

require 'zip/zip'

def open_wr fn
elts = fn.split("/")
dir = ""
elts[0...-1].each{|d|
raise RuntimeError, "Invalid filename '#{fn}'." if
d == "" or d == ".."
dir << d << "/"
Dir.mkdir dir unless File.directory? dir
}
raise RuntimeError, "File exists '#{fn}'." if
File.exist? fn
File.open(fn,"wb"){|f| yield f}
end

Zip::ZipInputStream:pen(ARGV[0]) { |io|
while (entry = io.get_next_entry)
STDERR.puts entry.name
# directories are of type file, they just have a trailing slash
next if entry.name[-1] == '/'[-1]
open_wr(entry.name){|out| out << io.read}
end
}

 

[Michal Suchanek]

.gz and .tar.gz files aren't hard to deal with since a working Ruby
1.8.5 is a prerequisite. Look at minitar, and the zlib library is part
of a working Ruby 1.8.5 install.

Yes, it just does not handle bzip2 which is used to compress the mingw
base that contains tar.

So you still need bzip2, and you can as well include a full
tar/gzip/bzibp2 suite then as it will probably make the decomression
somewhat faster and less memory intensive. It might waste a bit of
space as the minitar source would probably compress a few kilobytes
smaller than the tar binary. Didn't do any benchmarks, though.

Thanks

Michal

 

[James Tucker]

7za.exe from the 7z distribution is both open source, and really small.

Whilst it may not be a 'normal' tool, it very happily deals with all
of these issues in a single app, and common command line interface.

It's also pretty swift.

 

[Luis Lavena]

7za.exe from the 7z distribution is both open source, and really small.

Whilst it may not be a 'normal' tool, it very happily deals with all
of these issues in a single app, and common command line interface.

It's also pretty swift.

Until 'we' switch to other alternatives, either pure-ruby or externals
tools, I've uploaded a zip file which contains the needed files to get
you running:

http://code.mmediasys.com/installer3/deps.zip (320KB)

unzip into a new folder and put it in the PATH:

Example:

SET PATH=%PATH%;C:\installer3\dependencies

Then you can use rake without worring about how or here to get these
files.

Regards,

 

[Luis Lavena]

[...]

Until 'we' switch to other alternatives, either pure-ruby or externals
tools, I've uploaded a zip file which contains the needed files to get
you running:

[...]

No more!

(bumping this).

Thanks to Dennis Ranke Installer3 nows uses a pure-ruby solution to
download the required extracting tools using RubyZip code.

There is no need to install rubyzip gem!

You can grab exported code (revision 36) at the same URL:

http://code.mmediasys.com/installer3/latest.zip

Or checkout your own brach using bazaar:

bzr branch http://code.mmediasys.com/installer3/dev my-branch-feature

For those wanting to contribute, I'll be pushing further progress of
this project on rubyinstaller-devel mailing list:

http://rubyforge.org/mailman/listinfo/rubyinstaller-devel

Remember that the idea behind this is bootstrap the new One-Click
Installer for Windows.

Looking forward for contributors.

Regards,