Ruby-esque Inversion of Control

[Jamis Buck]

First, let me just say THANK-YOU to everyone who gave me feedback on my
presentation on Sunday. Rich Kilmer's suggestion that I serialize to
Ruby code, and various people (Austin, and various others that I can't
recall immediately) that suggested I ditch the YAML configuration file,
and Nathaniel Talbott who inadvertantly gave me some inspiration during
his test/unit presentation...

I've spent the last couple of days since my presentation thinking, and
thinking, and thinking. I'm going to do a brain dump here, and since I
know some of you have opinions on this matter, I hope you'll take a
moment to let me know if you think I'm on the right track to ruby-izing
Copland.

First of all, I realized that configuration points are just services
that implement a Hash-like or Array-like interface. Thus, there is no
really compelling need to treat them differently. In fact, implementing
them as service points allows you to use your own custom Hash-like or
Array-like class instead of Hash or Array.

Second, I think that Rich's suggestion of a Ruby-based domain-language
is very applicable, and ties in nicely with the inspiration I got from
Nathaniel's presentation. (Rich, any comments on the "domain language"
demonstrated below would be appreciated.)

Putting it all together, I'm thinking along the lines of the following
code snippet for defining service points:


registry = Copland::Registry.construct do

package "calc" do

service_point "Adder", Calc::Adder
service_point "Subtractor", Calc::Subtractor
service_point "Divider", Calc:ivider
service_point "Multiplier", Calc::Multiplier

service_point "Operations", Hash

service_point "Calculator" do
factory service { "copland.BuilderFactory" }
implementor do
klass Calc::Calculator
parameters [
service { "calc.Operations" }
]
end
end

with "calc.Operations" do |ops|
ops[:add] = service { "calc.Adder" }
ops[:subtract] = service { "calc.Subtractor" }
ops[:multiply] = service { "calc.Multiplier" }
ops[:divide] = service { "calc.Divider" }
end

end

end

calc = registry.service( "calc.Calculator" )

p calc.add( 8, 5 )
p calc.subtract( 8, 5 )
p calc.divide( 8, 5 )
p calc.multiply( 8, 5 )

It's still more verbose than I would like it, and there are problems
remaining that the above does not solve, but as a first pass, I think it
looks pretty nice. (Note that the "service {...}" syntax is necessary so
that the string is interpreted as a service reference, and not as a
string literal).

For those of you that were vocal in your opposition to the external
configuration file--how does the above strike you? What do you
like/dislike about it?

Other ideas:

* Allow "package" to be optional. If omitted, service points are added
to the global namespace.

* Allow true nesting of namespaces, for hierarchical package definitions.

Anyway, just getting my thoughts out. Comments would be greatly appreciated!

- Jamis

 

[Paul Brannan]

I still don't think that Copland is in any way useful to me (I just
simply don't have large trees of dependencies within a single process,
and where I do have dependencies, I handle it by instantiating all my
objects before the main loop; objects that depend on other objects
require them as parameters to their constructor).

That said, Copland does looks pretty clean and well-documented, and I
like the idea of making it even cleaner by eliminating the YAML file.
I've thought a lot about a library for the declarative syntax that you
and Nathaniel mentioned. I've written such a library, but it's
unfortunately not nearly as clean as I would like:

http://rubystuff.org/scriptable_config/scriptable_config.rb.html

It's used like this:

module MyConfigurationTemplate
extend ScriptableConfig::Template
Template = {
:host => Option(),
ort => Option(),
lugin => SectionTable ({
:id => Option(RequireType(Numeric)),
:file => Option(RequireType(String))
})
}
end

c = Config.new(MyConfigurationTemplate::Template)
c.read_file("config.rb")
c.validate()

Then the configuration file (config.rb) can look like this:

host "foo.com"
port 42

plugin('spam filter') {
id 1
file 'antispam.rb'
}

plugin('keepalive monitor') {
id 2
file 'keepalive.rb'
}

and produces a nested hash table with the specified configuration. I
never bothered to release the library because 1) when I asked about it on
irc, I was told that I was crazy and that I should use XML for this and
2) it still seems a bit klunky to use (though I am using it in
production -- it's been really valuable in a pinch to have the
configuration file be a ruby script, as this allows pieces of the
configuration file to be generated through connections to a database or
through reading an external config file, all without changing any of the
source code. it's a two-edged sword, though; reading configuration
files generally shouldn't have side effects, and this solution allows
them).

Anyway, it would be really cool if you could take this concept one step
further with Copland and make it even less verbose and more generically
useful. I'm not entirely sure what that would look like, as I am
somewhat limited by what I've already created (as David discussed in his
Rails talk).

Paul

 

[Richard Lyman]

From what I understand of Copeland, it's not really that useful to the
first developer - or better said, to the first time something is
developed since þere needs to be a lot of 'extra' work done - but it
pays off later...

When you or another coder goes back to that project and wants to use
something that was previously coded that it would be _very_ useful,
since with minimal changes you could substitue one part of your code
for a different, or better part.

Say I wrote something that used 10 parts. I left it for a while, and
later came back because I knew I could rewrite part #4 better. I
wouldn't have to mess with parts 1-3, or parts 5-10. Just make sure
part 4 quacks like the duck that it used to be and change a few lines
in Copeland and you're in business.

-Rich

 

[Paul Brannan]

Say I wrote something that used 10 parts. I left it for a while, and
later came back because I knew I could rewrite part #4 better. I
wouldn't have to mess with parts 1-3, or parts 5-10. Just make sure
part 4 quacks like the duck that it used to be and change a few lines
in Copeland and you're in business.

What you are describing (duck typing) seems to me totally orthogonal to
Copland. I can easily write my code like this:

class Part1
def initialize(part4)
@part4 = part4
end
end

class Part4
end

p4 = Part4.new
p1 = Part1.new(p4)

then come back later and change Part4 to something else and Part1 never
knows the difference.

As Jamis pointed out in his talk, Copland is useful when you have many
layers of dependencies and don't want to worry about how everything fits
together and gets initialized, but I just don't typically have code that
fits that description (and for the dependencies I do have, I find
explicitly representing them through parameters to the constructor, as
above, to be good enough for many situations, without resorting to a
more complex solution).

Anyway, my point was that while Copland isn't particularly useful to me
(the bulk of the code I write is in C++, not Ruby), I'm nevertheless
interested in its development.

Paul

 

[vruz]

As Jamis pointed out in his talk, Copland is useful when you have many

layers of dependencies and don't want to worry about how everything fits
together and gets initialized, but I just don't typically have code that
fits that description (and for the dependencies I do have, I find
explicitly representing them through parameters to the constructor, as
above, to be good enough for many situations, without resorting to a
more complex solution).

Sometimes lightweight IoC is also useful in highly componentized apps
such as IDEs.
Apart from Copland, Jamis has also proposed another, lighter approach
to IoC as an RCR, to be included with Ruby, and that's probably more
suitable for environments where Copland feels like an overkill.
Jamis called his reference implementation "pubsub", and I added some
comments to this (hopefully) useful RCR at:

http://rcrchive.net/rcr/RCR/RCR198

cheers,

 

[Jamis Buck]

What you are describing (duck typing) seems to me totally orthogonal to
Copland. I can easily write my code like this:

Not exactly... Copland is actually based strongly on the fact that Ruby
uses duck typing. It depends on that feature--thus, the two aren't
really orthogonal. Although you can do duck typing without Copland, you
can't do Copland without (at least some) duck typing.

class Part1
def initialize(part4)
@part4 = part4
end
end

class Part4
end

p4 = Part4.new
p1 = Part1.new(p4)

then come back later and change Part4 to something else and Part1 never
knows the difference.

Indeed, Paul -- you're using dependency injection here, without even
knowing it. You're just doing manual dependency injection, without
the benefit of any automation. For shallow dependency trees (typically
only one or two levels deep), this works fine, and is going to be more
efficient (though perhaps slightly less maintainable) than using a
container like Copland. However, when you ramp the dependency tree to
another level or two, or three, or four, it quickly becomes impossible
to do manual dependency injection in any reasonable fashion. At that
point, you'll either engineer your own solution, or take advantage of a
package that does it for you (like Copland).

As an example, I started reworking Net::SSH to take advantage of
Copland. I've only refactored a small fraction of it (maybe 10-15%), and
I'm already dealing with dependencies 3 or 4 levels deep. The original
implementation used either abstract factories, or just tightly coupled
the components together (one big reason why Net::SSH lacks a decent
suite of unit tests).

Anyway, my point was that while Copland isn't particularly useful to me
(the bulk of the code I write is in C++, not Ruby), I'm nevertheless
interested in its development.

Glad to hear you haven't given up on it.

Thanks, Richard and Paul, for your feedback!

- Jamis

 

[Charles O Nutter]

I forget who made this point at RubyConf, but the best example of why
IoC and Copland are good things is the idea of a Black Box.

It's not generally good practice if you need to go back in and modify
code to simply reconfigure something. In a larger application, you
could conceivably need to have multiple configurations for multiple
environments that require different implementations of components. You
need to be able to treat a system as a black box; in this case, a
pluggable system with multiple compatible implementations of
subservices and dependencies. Modifying code for each deployment is
not scalable.

For example, what if some service was distributed as a gem file? You
would not want to be mucking about in the deployed gem contents to
reconfigure it, would you? Ideally, for any given use of that service,
you could have an appropriate configuration for your target
environment. You might want to log to a file on one system or two
Windows event logs on another. You might want to have a DBM-based
persistence mechanism or use something beefier backed by a large
ORDBMS. Making these changes shouldn't ever require modifying code.

The debate over whether YAML is the best way to handle this
configuration is a whole separate matter. Coming from a Java world,
YAML is a considerable improvement over the reams of XML descriptors
I'm used to, so I'm probably too biased to make a good
ruby-appropriate recommendation

- Charlie

 

[Jim Weirich]

I forget who made this point at RubyConf, but the best example of why
IoC and Copland are good things is the idea of a Black Box.

I've added my attempt at explaining DI/IoC at http://onestepback.org. I
would be very interested in getting feedback on the effectiveness of the
article.

Thanks.

 

[Brian Candler]

I've added my attempt at explaining DI/IoC at http://onestepback.org. I
would be very interested in getting feedback on the effectiveness of the
article.

I was completely unaware of DI/IoC until now, so hopefully I'm a good
candidate I'd say it's an excellent document.

I immediately recognised the Service Locator pattern, which I've been using
without knowing it as such. To build the webapp that you describe, rather
than building the construction knowledge into Webapp.initialize, I would
build an outer skeleton 'runner' which instantiates all the objects and
links them together:

#!/usr/local/bin/ruby
require 'mywebapp'
logger = Logger.new
database = Database.newlogger=>logger)
error_handler = ErrorHandler.newlogger=>logger)
quotes = StockQuotes.newlogger=>logger,:error_handler=>error_handler)
authenticator = Authenticator.new(
:logger=>logger,:error_handler=>error_handler,:databbase=>database
)
webapp = Webapp.new(
:logger=>logger,:error_handler=>error_handler,:databbase=>database,
:quotes=>quotes,:authenticator=>authenticator
)
webapp.run

Clearly, these option hashes are similar to your Service Locators, but I'm
not tied to using them; if a particular object requires (logger,database) as
separate parameters, I just pass them in. I prefer hashes as it allows items
to be optional, e.g. pass in a logger or not.

In practice, there tends to be multiple versions of the skeleton. I might
have one version which runs under fastcgi, and another which starts a
webrick server, for example. Or I might decide to put the one of the objects
on a different machine, and access it via DRB. So the main problem I have is
that I end up with lots of very similar skeletons for different uses (unit
testing, command-line runner, fastcgi, webrick etc), so if the plumbing
needs to change, I have to remember to change each of the skeletons
accordingly.

DI would still require this plumbing, so I'm not sure it helps with this
particular problem, except perhaps I could put some of the invariant parts
of the DI skeleton into a library.

You point out as a limitation of the SL approach:

Also, Suppose both StackQuotes and Datebase found their loggers using
:logger, but we want to give them separate logger instances for some
reason. The explicit dependence on the name of the logger service makes
this a bit difficult.

I don't have this problem; I just write

logger1 = Logger.new
logger2 = Logger.new
database = Database.newlogger=>logger1)
error_handler = ErrorHandler.newlogger=>logger1)
quotes = StockQuotes.newlogger=>logger2,:error_handler=>error_handler)

So the main advantages I can see for the DI approach are:
1. When object creation must be dynamic, i.e. you can't pre-plumb
everything [I presume you could pass the DI object into the application,
and let it create things on demand later];
2. When you don't want to think about the creation order to satisfy
dependencies.

Is that a fair summary, or are there other advantages?

Finally, one thing I don't understand. In your final example, using DI in
your method "def create_application", I see all the dependencies registered
but not actually something which say "create me an :app and run it". Is
there something missing? Or does the DI framework simply try to build one of
every registered object in the dependency tree?

Regards,

Brian.

 

[Jim Weirich]

I was completely unaware of DI/IoC until now, so hopefully I'm a good
candidate I'd say it's an excellent document.

I immediately recognised the Service Locator pattern, which I've been using
without knowing it as such. To build the webapp that you describe, rather
than building the construction knowledge into Webapp.initialize, I would
build an outer skeleton 'runner' which instantiates all the objects and
links them together:

[... code elided ...]

Interesting. You are effectively creating a separate service locater
for each component. As you demonstrate, there are an infinite number of
variations in these patterns.

In practice, there tends to be multiple versions of the skeleton. I might
have one version which runs under fastcgi, and another which starts a
webrick server, for example. Or I might decide to put the one of the objects
on a different machine, and access it via DRB. So the main problem I have is
that I end up with lots of very similar skeletons for different uses (unit
testing, command-line runner, fastcgi, webrick etc), so if the plumbing
needs to change, I have to remember to change each of the skeletons
accordingly.

DI would still require this plumbing, so I'm not sure it helps with this
particular problem, except perhaps I could put some of the invariant parts
of the DI skeleton into a library.

One of the variations I discussed with Jamis included the possibility of
layering DI containers with namespaces. If I understand the concept
correctly, you configure the base system in a root namespace and then
layer the variations in namespaces on top of the root. If a service
isn't located in the specific namespace for the (lets say) Fast CGI
version, then it uses the services configured in the root. This would
reduced the duplication in the system.

Another approach is to use a single DI container, but configure it with
in different methods. One method could populate the static portion of
the application, and then different methods could populate it according
to whether it was a Fast CGI configuration or a webrick configuration.

So the main advantages I can see for the DI approach are:
1. When object creation must be dynamic, i.e. you can't pre-plumb
everything [I presume you could pass the DI object into the application,
and let it create things on demand later];

That's one approach. It does tie your system into a particular DI
framework at that point. If you want it to remain independent, then
just configure the DI container with a factory for the dynamic object
creation.

For example, once I built a system that could generate events in our
middleware product. All I needed to do was register the class of the
events that I wanted to create, eg.

container.registerevent_class) { MiddlewareEvent }

And then later, use that class to create the events.

event_creator = container.event_class
...
event_creator.new

This should also work with a service locater approach as well.

Finally, one thing I don't understand. In your final example, using DI in
your method "def create_application", I see all the dependencies registered
but not actually something which say "create me an :app and run it". [...]

Hmmm ... I was so concerned about getting the little pieces right I
missed the big one. That function probably should have ended by
returning: container.app

 

[Brian Candler]

One of the variations I discussed with Jamis included the possibility of
layering DI containers with namespaces. If I understand the concept
correctly, you configure the base system in a root namespace and then
layer the variations in namespaces on top of the root. If a service
isn't located in the specific namespace for the (lets say) Fast CGI
version, then it uses the services configured in the root. This would
reduced the duplication in the system.

Another approach is to use a single DI container, but configure it with
in different methods. One method could populate the static portion of
the application, and then different methods could populate it according
to whether it was a Fast CGI configuration or a webrick configuration.

Or maybe it could be as simple as allowing registrations to be overridden;
have a version of #register which does not check for duplicates.

def reregister(name, &block) # or have parameter override=false ?
@services[name] = Service.new(block, self)
end
def unregister(name)
@services.delete(name)
end

and in the application:

require 'myapp/di'
di = Myapp.dependencies
di.reregisterlogger) { |c| ...something different... } # override
di.app.run

It doesn't work for all cases - e.g. where you want to change logging so
that one component gets a different logger. In that particular example, you
could rebuild the dependency information to allow for it:

container.registerdefault_logger) { |c| Logger.new(c.logfilename) }
container.registereh_logger) { |c| c.default_logger }
container.registerdb_logger) { |c| c.default_logger }
container.registererror_handler) { |c| ... errh.logger = c.eh_logger ... }

But I wonder if the DI framework could handle it more intelligently,
allowing you to register a different Proc depending on the type of object
being created.

container.registerlogger) { |c| Logger.new("test1") }
container.registerlogger,:error_handler) { |c| Logger.new("test2") }

The second line is "when creating a :logger for an :error_handler, do this".
The first is "when creating a :logger at any other time, do that". Just a
thought; it breaks the idea that c.logger always gives an instance of the
same object though.

Cheers,

Brian.

 

[Jamis Buck]

It doesn't work for all cases - e.g. where you want to change logging so
that one component gets a different logger. In that particular example, you
could rebuild the dependency information to allow for it:

[snip]

In Copland, I have a LogFactory service which intelligently manages the
instantiation of logger instances. It can be configured to specify a
differently-configured logger for different services. This requires that
some additional information be passed to the constructor block, such as
the metadata about the service being constructed, so that the LogFactory
can be given enough information to configure the new logger correctly.

Something (roughly) like this:

container.register( :service ) { |c,meta|
svc = NewService.new
svc.logger = c.log_factory.get( meta )
svc
}

- Jamis

 

[James Britt]

I've added my attempt at explaining DI/IoC at http://onestepback.org. I
would be very interested in getting feedback on the effectiveness of the
article.

Early on, you refer to Dependency *Inversion*, rather than Inversion of
Control or Dependency Injection (which is what I think you meant to say).



James