Including other people's Ruby libs in your package

[James Britt]

I'm writing some code that depends on MIME::Types, and works better with
(but does not depend on) BlueCloth.

I'd like to package my code as both a Gem and as a more conventional
"install using cp" bundle.

What sort of project directory layout might work best?

Gem will go fetch dependencies, so I think I'm covered there (I can
leave these other files out of the gem and list them as dependencies),
but what might be some good installation techniques when not using a
Ruby gem?

I'm thinking now of putting the third-party Ruby files into a local
subdirectory of the installed code, and munging the load path in my
code, prior to calling require, to add this local directory. If the
user already has mime/types.rb, great, the global version gets called.
Otherwise 'require' should be able to find the copy installed in the
relative directory.

Thoughts?


James

 

[Aredridel]

I'm thinking now of putting the third-party Ruby files into a local

subdirectory of the installed code, and munging the load path in my
code, prior to calling require, to add this local directory. If the
user already has mime/types.rb, great, the global version gets called.
Otherwise 'require' should be able to find the copy installed in the
relative directory.

Speaking as a packager, don't. Just document, clearly, the dependencies
and where to get them.

It's annoying to have to patch away the extra cruft when packaging a
package for distribution with a linux distro -- but that's the job --
make it as light as possible with all the features.

Ari

 

[James Britt]

Speaking as a packager, don't. Just document, clearly, the dependencies
and where to get them.

It's annoying to have to patch away the extra cruft when packaging a
package for distribution with a linux distro -- but that's the job --
make it as light as possible with all the features.

I've considered this, but find it potentially too annoying to the user.

I would prefer that people be able to just download a tarball, extract
it, and copy over a directory to the libs dir. And it just works.




James

 

[Ara.T.Howard]

What sort of project directory layout might work best?

i use this layout

a top level 'include' file. one versioned, one not. this allows me install
multiple versions and do like 'require "package-0.0.0" when i want to link to a
specific version. doing a 'require "package"' will always pick up the latest
greatest.

lib/package-0.0.0.rb
lib/package.rb

unless defined? $__package__
module Package
LIBNAME = "package"
VERSION = "0.0.0"
LIBVER = "#{ LIBNAME }-#{ VERSION }"
DIRNAME = File.dirname(File.expand_path(__FILE__)) + File::SEPARATOR
ROOTDIR = File.dirname(DIRNAME)
LIBDIR = File.join(DIRNAME, LIBVER) + File::SEPARATOR

require LIBDIR + 'a'
require LIBDIR + 'b'
end
$__package__ = __FILE__
end


actual package impl lives in a versioned directory (for same reason as above).
these files set alot of the same parameters as the include files. these vars
allow me to do relative requires and also allow mutual requies. this is
extremely nice for unit testing since each class can require independantly the
files it needs with no fear of double requires do the the 'require "a" -
require "a.rb"' bug.

lib/package-0.0.0/a.rb

unless defined? $__package_a__
module Package
LIBDIR = File.dirname(File.expand_path(__FILE__)) + File::SEPARATOR unless
defined? LIBDIR

require LIBDIR + 'b'

class A
end
end
$__package_a__ = __FILE__
end

lib/package-0.0.0/b.rb

unless defined? $__package_b__
module Package
LIBDIR = File.dirname(File.expand_path(__FILE__)) + File::SEPARATOR unless
defined? LIBDIR

require LIBDIR + 'a'

class B
end
end
$__package_b__ = __FILE__
end


this seems a pain but i have code generator scripts to do the work.

cheers.


-a
--
===============================================================================
| EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE :: 303.497.6469
| A flower falls, even though we love it;
| and a weed grows, even though we do not love it.
| --Dogen
===============================================================================

 

[Jamis Buck]

I've considered this, but find it potentially too annoying to the user.

I would prefer that people be able to just download a tarball, extract
it, and copy over a directory to the libs dir. And it just works.

On the other hand, it can be very annoying for a user as well when they
already have a *newer* version of that same dependency installed, and
then they install your library and suddenly things break.

If nothing else, provide two versions of your package: one *with*
dependencies, and one without. I'm in the camp that prefers applications
NOT be shipped with their dependencies included, but I can understand
that others may prefer the other way.

The moral of the story is that there isn't a perfect solution to this,
unfortunately. Rubygems' dependency feature comes close, though.

--
Jamis Buck
(e-mail address removed)
http://www.jamisbuck.org/jamis

"I use octal until I get to 8, and then I switch to decimal."

 

[James Britt]

On the other hand, it can be very annoying for a user as well when they
already have a *newer* version of that same dependency installed, and
then they install your library and suddenly things break.

That's right. So let's don't do that.

That question is, What's the best way?

One approach is change $: to look in some specified directory, as a last
resort, if the standard path does not turn up the needed lib.

So, for example, if BlueCLoth is already installed, then the
pre-packaged copy is never loaded because a "correctly" installed
version has been found.

If my lib is installed in /foo/bar and my class file does
$:.push "/foo/bar/dependencies" # or something along those lines
then my special directory should be the last place Ruby looks for a
file. Or no?

If nothing else, provide two versions of your package: one *with*
dependencies, and one without. I'm in the camp that prefers applications
NOT be shipped with their dependencies included, but I can understand
that others may prefer the other way.

The moral of the story is that there isn't a perfect solution to this,
unfortunately. Rubygems' dependency feature comes close, though.

For Gems, it might be nice (or feature-creep overkill) to specify that,
if a user refuses to install dependency Foo, then fail the installation,
but if they refuse pseudo-dependency Bar, that's OK, keep going with a
"You're going to miss some good stuff" warning.

 

[Chad Fowler]

For Gems, it might be nice (or feature-creep overkill) to specify that,
if a user refuses to install dependency Foo, then fail the installation,
but if they refuse pseudo-dependency Bar, that's OK, keep going with a
"You're going to miss some good stuff" warning.

I've had this on the list for a while, but I'm still not sure if it's
feature creep or a good feature to add. I have what I think is a
healthy uneasiness about adding anything to the gem spec, but
something like this may just find it way in.

Chad

 

[Austin Ziegler]

I'm writing some code that depends on MIME::Types, and works
better with (but does not depend on) BlueCloth.

What sort of application is it? (I'm curious, of course, as I wrote
MIME::Types

With Ruwiki, we depended on Algorithm:iff -- and we will be
depending on Diff::LCS -- and I will be packaging Diff::LCS with
Ruwiki. This is because Ruwiki is intended to be standalone, except
for the Ruby core libraries. Ruwiki ensures that its versions will
be loaded in preference to system versions by modifying $LOAD_PATH
such that the Ruwiki library directory is first.

If and when I make it "installable" rather than an "unpack and run"
program, I will do further work with the install.rb such that I do a
"require 'diff/lcs'" (and any other dependencies); if it is not
found, then I will install Diff::LCS. If it is found, then I will
check to make sure that it is at least version 1.1. If not, then I
will prompt the user for replacement. Now, the real problem is that
Ruwiki -- as of right now -- only includes those parts of Diff::LCS
which are necessary to run Ruwiki. Thus, if I take that route, I
will probably offer multiple packages.

An alternative -- and perhaps better way for the .tar.gz concept --
is to provide the required and optional packages as .tar.gz packages
within one of your distribution packages. This means that
OS-specific packagers don't have to strip out the code by hand, and
you don't have to worry about those tests.

bin/
dep/required/mime-types-1.13.tar.gz
dep/optional/bluecloth-0.3.2.tar.gz
...

That said, if you have a standalone situation as we have with
Ruwiki, feel free to include MIME::Types or any of my other
packages. I have RubyForge projects for MIME::Types, Text::Format,
and Transaction::Simple, now -- I just haven't announced them.

-austin

 

[James Britt]

I've had this on the list for a while, but I'm still not sure if it's
feature creep or a good feature to add. I have what I think is a
healthy uneasiness about adding anything to the gem spec, but
something like this may just find it way in.

An idea I floated on another list (and somewhat off-topic for the main
discussion there at the time) was having some way to vet code before
automatically installing it.

Run a set of sanity-check tests over the proposed code (for example,
look for references to $SAFE, alterations of base classes,
upgrades/downgrades of existing files, and the use of camelCase for
method names), then report a score of some sort.

The user can then decide that maybe the installation isn't a good idea,
or it presents some risk, or would fill the hard drive, and cancel the
installation.

There have been a number of times I installed Perl code using the CPAN
shell, astounded by the number of extra libs that were installed as
dependencies. I thought I had a new screen saver.

Is there some simple way to plug in additional operations along the
RubyGems installation process path, such that people can write a class
and tell RubyGems, "When you get to this step, run this code, passing in
the directories of the extracted but not-yet-installed gems"?

Then write an XML (oops, I mean YAML) file that defines a customized
processing path.

Or something. But then folks who wanted to embark on feeping creaturism
could have a field day.



James

 

[James Britt]

What sort of application is it? (I'm curious, of course, as I wrote
MIME::Types

Well, if you *must* know, it takes either a string of XHTML or
plain-text and produces a MIME multipart/alternative message body. The
code looks at the given string and automagically creates the alternative
rendering (given, XHTML, it also creates a plain-text version, and vice
versa. Works best when handed a BlueCloth string, but the
plain-to-peanuts conversion is configurable).

If the resulting (or given) XHTML contains local references to images,
they are encoded and added to the message body as multipart/related; the
img src value is replace with a cid: ref to the encoded image segment.

MIME::Types is used to figure out the MIME type of the image.

With Ruwiki, we depended on Algorithm:iff -- and we will be
depending on Diff::LCS -- and I will be packaging Diff::LCS with
Ruwiki. This is because Ruwiki is intended to be standalone, except
for the Ruby core libraries. Ruwiki ensures that its versions will
be loaded in preference to system versions by modifying $LOAD_PATH
such that the Ruwiki library directory is first.

I've got a similar deal with Blogtari (also stand-alone, with no
assumptions about user permissions other than their on httpdocs or
public_html directory), which includes RedCloth and BlueCloth, but it
will try to first load them from the standard path.

If and when I make it "installable" rather than an "unpack and run"
program, I will do further work with the install.rb such that I do a
"require 'diff/lcs'" (and any other dependencies); if it is not
found, then I will install Diff::LCS. If it is found, then I will
check to make sure that it is at least version 1.1. If not, then I
will prompt the user for replacement. Now, the real problem is that
Ruwiki -- as of right now -- only includes those parts of Diff::LCS
which are necessary to run Ruwiki. Thus, if I take that route, I
will probably offer multiple packages.

Blogtari has its own installation routine. Everything is packed up
using a hacked version of Tar2Rubyscript (quite slick); there's code in
there to prompt the user for configuration details, as well as try to
managed upgrades without stomping existing user config data or
templates. But nothing to verify available dependencies.

An alternative -- and perhaps better way for the .tar.gz concept --
is to provide the required and optional packages as .tar.gz packages
within one of your distribution packages. This means that
OS-specific packagers don't have to strip out the code by hand, and
you don't have to worry about those tests.

bin/
dep/required/mime-types-1.13.tar.gz
dep/optional/bluecloth-0.3.2.tar.gz
...

That's a really good idea. I think my biggest gripe with packages with
multiple dependencies is that they send you off on a series of hunt and
fetch operations. (This is *especially* annoying if you've loaded your
laptop with assorted libs to examine for a presentation you plan on
writing on the plane ride to a conference. Or so I hear.)

So simply bundling the required libs in such a way that the user at
least has them to install is a good compromise. And I like the
directory tree.

That said, if you have a standalone situation as we have with
Ruwiki, feel free to include MIME::Types or any of my other
packages. I have RubyForge projects for MIME::Types, Text::Format,
and Transaction::Simple, now -- I just haven't announced them.

Sweet. Thanks.


James