why can't Ruby load .gem files directly?

[Joshua Haberman]

I've just been reading the recent threads about gems. I don't have
much background on this issue, but I have a few questions/concerns
about the current state of gems.

My primary question is: why do I have to "install" a gem? What about
the gem design would make it fundamentally impossible for Ruby to
load a .gem file directly?

Think how great it would be if "installing a gem" was equivalent to
"putting the gem file in your RUBYLIB path." What could be simpler?
Sure, you couldn't put checks in place to make sure I've installed
all the right dependencies, but you'll figure that out at runtime,
when I try to load that gem (or you could use "gem check").

My concern with the current design is that it is too inflexible. It
imposes a model where you have to use the "gem" command to do
anything, and you have to hope that your use case is one of the
operations that "gem" supports. For example, can gems support these
use cases?

- I want to try out a gem on a system where I don't have root. (or
where I don't want to install the software globally yet)

- I want to have more than one local gem repository (eg. /usr/lib
vs. /usr/local/lib), and I want dependencies to be fulfilled across
repositories (eg. a gem in /usr/lib can have a dependency be
fulfilled by a gem in /usr/local/lib)

- I want to install a gem from a gemfile I already have (ie: don't
fetch it from the network)

- I want to "unpack" a gem into a directory structure that can be
deployed to another machine, without having to run "gem" on the
remote machine (and without updating any global index on my current
machine)

- same as the previous one, but I also want to "unpack" the gem into
a different directory root than where it will live on the remote
machine. I may not even know what directory it will live in on the
remote machine.

What I don't like is the feeling that the "gem" command is a
gatekeeper between you and your system -- you have to go through it
to get anything done. And it hides the details of what's going on,
making the system very opaque. It was difficult for me to figure out
which of the above use cases are supported, because the rubygems
documentation doesn't tell you the details of what "gem" does.

If Ruby could load .gem files directly, you would immediately support
any use case, because they would be as flexible as single .rb files:

- "cp foo.gem /path/to/my/RUBYLIB" could be used in place of "gem
install" (you could still use "gem install" to query remote servers
if you want)

- "rm foo.gem" could be used in place of "gem uninstall"

- "ls foo*.gem" could be used in place of "gem list"

I think this would make the system more flexible and transparent. I
would argue that any properties of the gem format that make this
impossible should be reconsidered.

Josh

 

[Jim Weirich]

[...] For example, can gems support these use cases?

- I want to try out a gem on a system where I don't have root. [...]
Yes.

- I want to have more than one local gem repository [...]
Yes.

- I want to install a gem from a gemfile I already have [...]
Yes.

- I want to "unpack" a gem into a directory structure that can be
deployed to another machine, without having to run "gem" on the
remote machine
Yes.

- same as the previous one, but I also want to "unpack" the gem into
a different directory root [...]

Yes.

 

[Jim Weirich]

If Ruby could load .gem files directly, you would immediately support =A0
any use case, because they would be as flexible as single .rb files:

Actually, this was strongly considered in early versions of gems. It wou=
ld=20
require a require hack that is much more intrusive than the current one=20
RubyGems uses now. The real stopper was that we didn't see an easy way t=
o=20
support shared libraries (.so's and .dll's) on all the different platform=
s.

--=20
-- Jim Weirich (e-mail address removed) http://onestepback.org

 

[Ryan Leavengood]

Think how great it would be if "installing a gem" was equivalent to
"putting the gem file in your RUBYLIB path." What could be simpler?
Sure, you couldn't put checks in place to make sure I've installed
all the right dependencies, but you'll figure that out at runtime,
when I try to load that gem (or you could use "gem check").

My original prototype for RubyGems did exactly this. I considered gem
files to be much like a Java jar file, and the code was loaded
directly from the gem when required. My prototype used some custom
file formats (two actually, a plain text and a binary one, just for
fun) that contained the embedded code and some metadata. I presented
this prototype at RubyConf 2001, but stopped development due to lack
of certain libraries (zip for example) as well as some other real-life
issues (like being overwhelmed, losing interest and not having time.)

When Jim and the other guys decided to bring RubyGems back to life at
RubyConf 2003, they started with some fresh code, and I'm not sure why
they chose to implement gems as they did (but I'm sure there was a
good reason for having the separate install step, like maybe
performance.)

Either way, as Jim says your use cases are supported in the latest
RubyGems, though now that is has been brought up, I'd like to play
with some of my old ideas again.

Based on recent discussions though, I wonder if RubyGems need to be
re-written for a third time (though strictly speaking my first
revision really was a prototype and didn't have even close to the
feature-set of the current system.)

Ryan

 

[Joshua Haberman]

--Apple-Mail-2--403818632
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
charset=US-ASCII;
delsp=yes;
format=flowed

Actually, this was strongly considered in early versions of gems.
It would
require a require hack that is much more intrusive than the current
one
RubyGems uses now. The real stopper was that we didn't see an easy
way to
support shared libraries (.so's and .dll's) on all the different
platforms.

I'm not sure what you mean by that. Could you elaborate (or point me
to documentation that provides more detail about this problem)?

Do gems attempt to be platform-independent even where there are C
extensions, and compile extensions at gem install time? That doesn't
sound like a very good idea.

If gems that have C extensions were kept platform-specific, there
shouldn't be a problem with shared libraries.

Josh

--Apple-Mail-2--403818632--

 

[Ryan Leavengood]

Actually, this was strongly considered in early versions of gems. It wou= ld
require a require hack that is much more intrusive than the current one
RubyGems uses now. The real stopper was that we didn't see an easy way t= o
support shared libraries (.so's and .dll's) on all the different platform=

s.

The shared library part makes sense. I didn't even get close to that
in my little prototype.

It seems that the interpreter itself would need to natively support
gem files and be able to load shared libraries from them directly
before you could get away from having the separate install step. It
would be interesting to have gems with C source that are compiled on
demand (given the right setup on the installation machine) and then
used directly. So for example you would start off with
binlib-1.0-src.gem and afterwards have the src gem and then
binlib-1.0.gem with the embedded shared libs. But maybe you guys on
the RubyGems team already considered all this and have several good
arguments against it.

Ryan

 

[Joshua Haberman]

Either way, as Jim says your use cases are supported in the latest
RubyGems, though now that is has been brought up, I'd like to play
with some of my old ideas again.

Please do!

Say that you had two implementations of gems with identical feature
sets, but one used the "gem" command to do everything and one let you
use .gem files directly. I would *much* prefer using the latter.

Though I am glad that Jim says my use cases are supported, I still
feel that having to interact with the system using an opaque command
like 'gem' is inferior to being able to interact with my filesystem
directly. It also will make me worry that in the future I'll run
into an unsupported use case I hadn't thought of.

Based on recent discussions though, I wonder if RubyGems need to be
re-written for a third time (though strictly speaking my first
revision really was a prototype and didn't have even close to the
feature-set of the current system.)

Yes, I don't know the details of the current controversy (ie. why
Debian is having a problem packaging gems), but standardizing on a
gem format that is at odds with OS packages seems like a very bad idea.

Josh

 

[Jim Weirich]

I'm not sure what you mean by that. =A0Could you elaborate (or point me= =A0
to documentation that provides more detail about this problem)?

If you have a shared library embedded in your gem file, how do you point =
your=20
LD_LIBRARY_PATH environment variable at it so that the OS knows about it.=
=20
Not only is a hard problem to solve, it is a hard problem that must=20
(potentially) be solved differently on each platform.

--=20
-- Jim Weirich (e-mail address removed) http://onestepback.org

 

[Michael Schoen]

- I want to try out a gem on a system where I don't have root. [...]

Yes.

How is this one done? In particular, how would I install a gem as a
non-root user, and have access to both my local gems and those that are
installed by root?

 

[Devin Mullins]

Say that you had two implementations of gems with identical feature
sets, but one used the "gem" command to do everything and one let you
use .gem files directly. I would *much* prefer using the latter.

Actually, it doesn't sound like they're mutually exclusive. Stick a
little flag in the metadata.gz that says "I don't have a compile step!"
and then it can be stuck on the $LOAD_PATH directly. Otherwise, upon
require-ing something from inside the gem, you'll get an error "Install
meeee!!"

Yes, I don't know the details of the current controversy (ie. why
Debian is having a problem packaging gems)

Neither do I. Are they trying to repackage gems as... err.. whatever the
apt-get file extension is? If so, why?

Devin

 

[Eivind Eklund]

Actually, it doesn't sound like they're mutually exclusive. Stick a
little flag in the metadata.gz that says "I don't have a compile step!"
and then it can be stuck on the $LOAD_PATH directly. Otherwise, upon
require-ing something from inside the gem, you'll get an error "Install
meeee!!"


Neither do I. Are they trying to repackage gems as... err.. whatever the
apt-get file extension is? If so, why?

They're trying to repackage software that the author assume to be
Gem-distributed as Debian packages (.deb, IIRC). The reason for this
is simple: To provide Ruby software on Debian for Debian users in the
way Debian users are used to. These users do not particularly care
about Ruby, and definately do not want to have to know one packaging
system for handling Ruby software, and another for handling Perl, and
another for handling Java, and another for handling Haskell, and
another for handling Python, and another way for handling C code, and
...

Instead, they want they software to Just Work Like All Other Software,
the way they are used to, and to not have to care about the language
the software is written in at all.

The problem is when Gems provide guarantees to the authors that are in
violation of how Debian users are used to software working, and no way
to implement things the authors need in a way that is possible to use
with the way Debian users want. Specifically, there's a problem with
the the-software-is-in-a-single-directory assumption.

Eivind.

 

[Corey Lawson]

------=_Part_125_24618124.1128270070937
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

It sounds like a debian user problem, not a Ruby user problem. I think most
Ruby people (who probably are not running ruby on Debian) want Ruby to work
like...Ruby, whether it's on a Mac, Windows, RedHat, Debian, Slolaris,
whatever. Has a similar beef been made about Java on Debian? (Why must Java
apps be deployed as .jar, .ear, war, whatever. Can't they pick just one and
make it work with apt-get?).

They're trying to repackage software that the author assume to be
Gem-distributed as Debian packages (.deb, IIRC). The reason for this
is simple: To provide Ruby software on Debian for Debian users in the
way Debian users are used to. These users do not particularly care
about Ruby, and definately do not want to have to know one packaging
system for handling Ruby software, and another for handling Perl, and
another for handling Java, and another for handling Haskell, and
another for handling Python, and another way for handling C code, and
...

Instead, they want they software to Just Work Like All Other Software,
the way they are used to, and to not have to care about the language
the software is written in at all.

The problem is when Gems provide guarantees to the authors that are in
violation of how Debian users are used to software working, and no way
to implement things the authors need in a way that is possible to use
with the way Debian users want. Specifically, there's a problem with
the the-software-is-in-a-single-directory assumption.

Eivind.

------=_Part_125_24618124.1128270070937--

 

[Brian Schröder]

It sounds like a debian user problem, not a Ruby user problem. I think mo= st
Ruby people (who probably are not running ruby on Debian) want Ruby to wo= rk
like...Ruby, whether it's on a Mac, Windows, RedHat, Debian, Slolaris,
whatever. Has a similar beef been made about Java on Debian? (Why must Ja= va
apps be deployed as .jar, .ear, war, whatever. Can't they pick just one a= nd
make it work with apt-get?).

If it is a debian user problem that is sad, but I have to say that I
try to avoid any libraries that are not packed as a deb. I do not have
gems installed because the duality of the gem and the debian system is
annoying me.

This may be just an anecdote, but a lot of libraries are loosing at
least me as a user. I have packaged deb's myself and I don't think it
is a very complicated process, so I wonder why it is difficult to go
from .gem to .deb?

brian

 

[Kevin Brown]

[*snip*]

It sounds like a debian user problem, not a Ruby user problem. I think most
Ruby people (who probably are not running ruby on Debian) want Ruby to work
like...Ruby, whether it's on a Mac, Windows, RedHat, Debian, Slolaris,
whatever. Has a similar beef been made about Java on Debian? (Why must Java
apps be deployed as .jar, .ear, war, whatever. Can't they pick just one and
make it work with apt-get?).

It's both, because gems should be flexible enough to do both, especially since
doing so provides no particular problems to either group. That way, if my
mom wants to install a gem, she uses the debian package, and if I want to, I
do it the 'real' way.

Ultimately, if this results in an internet-schlong-waving-contest, everyone
loses. Debian also has the option of having a package type that includes a
wget and then a gem-install, but this is very contrary to the fact that once
you download the .deb package, you HAVE the software and that's IT.
Introducing nasty hacks like above mentioned are not good for Debian, and
that's why the debate is currently raging.

Also note that it's not just a Debian problem, but a problem of anyone who
uses .debs, including Ubuntu, Kubuntu, Mephis, and a whole host of other
distros.

 

[Austin Ziegler]

If it is a debian user problem that is sad, but I have to say that I
try to avoid any libraries that are not packed as a deb. I do not have
gems installed because the duality of the gem and the debian system is
annoying me.

This may be just an anecdote, but a lot of libraries are loosing at
least me as a user. I have packaged deb's myself and I don't think it
is a very complicated process, so I wonder why it is difficult to go
from .gem to .deb?

Because .deb has no concept of multiple versions installed
simultaneously being available for use. Because .gem doesn't have an
answer to the DATADIR problem (the only *real* problem with the
1-gem-1-dir solution, as everything else is simply SMOP and prefixes).
Anything else is ... going to become impolitic very quickly. If you
want to see a *long* rant by me about it, as well as proposals that
*should* be able to make this work even for Debian, read the ruby-core
threads about integrating RubyGems into Ruby core.

-austin

 

[Devin Mullins]

They're trying to repackage software that the author assume to be
Gem-distributed as Debian packages (.deb, IIRC). The reason for this
is simple: To provide Ruby software on Debian for Debian users in the
way Debian users are used to. These users do not particularly care
about Ruby, and definately do not want to have to know one packaging
system for handling Ruby software, and another for handling Perl, and
another for handling Java, and another for handling Haskell, and
another for handling Python, and another way for handling C code, and
...

Okay (and thanks for the reply). That's weird. If they don't paticularly
care about Ruby, what are they doing installing Ruby gems (usually
libraries)? If they're trying to install an application that happens to
be written in Ruby, then does it not suffice for the .deb to list ruby
and rubygems as dependences, and then for the install step to be `gem
install the-app-dependencies` or whatever?

Devin

 

[Devin Mullins]

but this is very contrary to the fact that once
you download the .deb package, you HAVE the software and that's IT.

Ah. Well, err, to qualify my previous post, couldn't the .deb just
include the .gem(s) inside, and then call `gem install thegem(s)`?

Devin

 

[Kevin Brown]

Okay (and thanks for the reply). That's weird. If they don't paticularly
care about Ruby, what are they doing installing Ruby gems (usually
libraries)? If they're trying to install an application that happens to
be written in Ruby, then does it not suffice for the .deb to list ruby
and rubygems as dependences, and then for the install step to be `gem
install the-app-dependencies` or whatever?

Nope, because in other debian-ish situations with libraries (be they written
in C and compiled) you can put them in and pull them out seperate of the
application by using apt-get, which makes a LOT of sense.

What would happen when you uninstall the program if packaged as you mention?
You can't remove the gems, and you can't really leave them there either...

 

[Kevin Brown]

Ah. Well, err, to qualify my previous post, couldn't the .deb just
include the .gem(s) inside, and then call `gem install thegem(s)`?

That was my first thought too. No, because if you distribute the debians as
part of a distro, you're requiring them to have an internet connection to get
them, and in fact there is no way at that point to include said software as
part of a distro that you can install without an internet connection.

Unless they set up a gems repository, and have it ask on the CD, but that gets
really nasty when you start realizing that this all works contrary to the way
apt-get has worked in the past.

Also, my mom (who just unplugged her network cable) wants to know what this
funky error about WGET FAILED FOR GEM INSTALL FROM DEB PACKAGE means. (I'm
pointing out that it doesn't allow users the precious luxury of ignorance,
which will be important if Linux ever wants to world dominate.)

 

[gabriele renzi]

Devin Mullins ha scritto:

Ah. Well, err, to qualify my previous post, couldn't the .deb just
include the .gem(s) inside, and then call `gem install thegem(s)`?

no, since gems make layout assumptions that do not match the debian
policy (nor the policy of most package systems on *nix, it seem).
Anyway, there was a huge thread on this on ruby-core that people may
like to read, since some of this.

I have faith in the rubygems dev crew, anyway, so that the problems can
be addressed before merging it in ruby's HEAD.