DI service change notifications (Syringe)

[leon breedt]

hi,

i'm playing around with Syringe (http://ruby.jamisbuck.org/syringe/),
and i've got an idea for additional behaviour that i want, though i'm
not sure if its a good idea yet.

what i want, is the option for services to be less static (i.e. not a
singleton where instantiation happens only once. i want more control
over service lifecycle..)

would what i'm asking for be a different "model", to use Syringe
terminology? the ability to flag a service as dirty, and
automatically, all its consumers get flagged as dirty too, for
refreshing on next usage.

contrived pseudocode example:

env = { :log_filename => 'test.log' }
container.registerlog_filename) { env[:log_filename] }
container.registerlogger) { |c| Logger.new(c.log_filename) }

now, since one can change the value in env, when making that change,
the matching :log_filename service would be flagged as dirty, and, on
next usage, the block executed again, and logger logs to different
file without program restart.

good idea? crap?

leon

 

[leon breedt]

contrived pseudocode example:

env = { :log_filename => 'test.log' }
container.registerlog_filename) { env[:log_filename] }
container.registerlogger) { |c| Logger.new(c.log_filename) }

i am aware that there will have to be a link between env and container
to ensure that the dirty flag will be set, first of all, and set on
the correct service, secondly

possibly env would be a special service instead of floating Hash, and
access to env would happen through an interceptor gatekeeper...

leon

 

[Jamis Buck]

hi,

i'm playing around with Syringe (http://ruby.jamisbuck.org/syringe/),

Wow. I'm surprised -- I haven't even made an announcement about it,
except on my blog. You realize, I hope, that Syringe is in a HUGE state
of flux right now, and the API is guaranteed to change in all kinds of
non-backwards-compatible ways, right?

env = { :log_filename => 'test.log' }
container.registerlog_filename) { env[:log_filename] }
container.registerlogger) { |c| Logger.new(c.log_filename) }

now, since one can change the value in env, when making that change,
the matching :log_filename service would be flagged as dirty, and, on
next usage, the block executed again, and logger logs to different
file without program restart.

good idea? crap?

A new service model would not really help in this case, because once the
Logger.new constructor is called, the log_filename has already been
returned as a string. What you need is to create an object that
represents the filename, without _being_ the filename, and have its
#to_str method defined to return the filename:

class LogFilename
def to_str
"test.log"
end
end

Then, you can register the :log_filename service to an instance of that:

container.register( :log_filename ) { LogFilename.new }

If you define LogFilename correctly, you could then change the string
that gets returned there, and have all consumers of the filename take
advantage of that.

But, there's a problem. In your example, you had Logger be the consumer
of the filename. Logger is not designed to allow the filename to be
changed. You would have to actually have a new logger instance be
created and substituted in place, something Ruby doesn't support without
using proxy objects (unless you're using evil.rb).

SO. What is needed is a way to allow consumers of consumers of
:log_filename (that is to say, consumers of :logger) to detect when the
:log_filename service was modified and then grab a new instance of :logger.

What is really needed, though, is an implementation of Logger that
allows the filename to be changed on the fly. If that existed, there
would be an easier (i.e., more tractable) solution.

 

[Jamis Buck]

contrived pseudocode example:

env = { :log_filename => 'test.log' }
container.registerlog_filename) { env[:log_filename] }
container.registerlogger) { |c| Logger.new(c.log_filename) }

i am aware that there will have to be a link between env and container
to ensure that the dirty flag will be set, first of all, and set on
the correct service, secondly

possibly env would be a special service instead of floating Hash, and
access to env would happen through an interceptor gatekeeper...

Arg. After my last rambling message, I realized I might have missed the
mark of what you were asking.

Did you want:

(A) The have existing references to Logger refresh themselves and
start logging to the new file? or

(B) To have existing references to Logger remain unchanged and have
new loggers start logging to the new file?

(A) is not so simple to do (as I mentioned in my last post). (B),
however, would be much more straightforward.

- Jamis

 

[leon breedt]

Did you want:

(A) The have existing references to Logger refresh themselves and
start logging to the new file? or

(B) To have existing references to Logger remain unchanged and have
new loggers start logging to the new file?

(A) is not so simple to do (as I mentioned in my last post). (B),
however, would be much more straightforward.

I'm not sure I was clear enough, and I am still of the opinion (A)
would not be *that* complex to implement, bear with me, I'm not that
good at describing it in English

I wanted the registry to keep track of the dependency graph, so that
when a service was changed, it would be marked dirty. The registry
would then walk the dependency graph in reverse order, and mark all
dependant services as dirty too. Being dirty just means that the next
time a request for a dirty service was made, the creation block
would be executed again, result cached, and marked clean.

I realize "service was changed" is rather vague.

In my case, I would probably write a service to contain configuration
values like "filename" (I don't like mixing data-only services with
services that actually do something). I'd add an domain-specific
interceptor wrapping this configuration service, that would know which
method invocations on the service are 'writes'. The interceptor would
then send a message to the registry, saying "service :config is
dirty". The registry, because it knows the dependencies on :config,
can mark :config and all things that used :config, as dirty.

The registry would know the dependency graph, because: any services
depending on :config would have done a "reg.config.value" in their
construction block.

In effect, this would be (A) above.

Pseudo-code:

reg.registerconfig) { Configuration.new }
reg.interceptconfig).with { |reg| ConfigurationInterceptor.new(reg} }
reg.registerlogger) { |reg| Logger.new(reg.config.filename) }

....

somewhere else in application: reg.config.filename = "newfile"


Because the construction block for :logger called reg.config, the
registry is aware that it depends on :config. The reg.config.filename
assign would have been caught by the interceptor as a "write", and
once completed, the interceptor would send a dirtyconfig) message to
the registry.

I was thinking, for this particular approach to work, |reg| in the
:logger construction block might need to be a proxy that collects the
services used in the block, and adds them to a dependency graph,
instead of being the registry itself (for thread-safety?)

Hope that gives you more of an idea of what I'm getting at...

Leon

 

[leon breedt]

You realize, I hope, that Syringe is in a HUGE state
of flux right now, and the API is guaranteed to change in all kinds of
non-backwards-compatible ways, right?

Understood

A new service model would not really help in this case, because once the
Logger.new constructor is called, the log_filename has already been
returned as a string.

Refreshing the dependendant services would take care of this
particular problem (more details in my other reply), and wouldn't
require the creation of stub classes for "data" services.

What is really needed, though, is an implementation of Logger that
allows the filename to be changed on the fly. If that existed, there
would be an easier (i.e., more tractable) solution.

The weakness of that is that it requires an API change to Logger,
which would not be required if dependency graph + dirty flag
propagation existed (other post). In addition, you'd get things like
automatic closing (as the previous logger goes out of scope) and
re-opening of the new file for free, without any Logger changes as
well.

However, it might be, that in some cases, service creation is
heavyweight enough that this "push" model is whats required. Or, as in
the example in my other post, if something changes an unrelated value
in the :config store, you don't want all services to be refreshed. In
which case it probably makes sense to have scoped :config* stores for
all configuration pertaining to a particular set of services.

Leon

 

[leon breedt]

In addition, you'd get things like
automatic closing (as the previous logger goes out of scope) and
re-opening of the new file for free, without any Logger changes as
well.

Goes without saying that for this to work, all classes must use
reg.logger instead of caching a logger reference

Leon

 

[Jamis Buck]

Pseudo-code:

reg.registerconfig) { Configuration.new }
reg.interceptconfig).with { |reg| ConfigurationInterceptor.new(reg} }
reg.registerlogger) { |reg| Logger.new(reg.config.filename) }

....

somewhere else in application: reg.config.filename = "newfile"


Because the construction block for :logger called reg.config, the
registry is aware that it depends on :config. The reg.config.filename
assign would have been caught by the interceptor as a "write", and
once completed, the interceptor would send a dirtyconfig) message to
the registry.

Hmmm. But there's still the issue of the reg.config.filename invocation
(to use the example above) knowing the context in which it is being
called. It would need to know, specifically, which service point
logger) was being instantiated, so that it could relate the dependency
back to that service... hmmm.

I was thinking, for this particular approach to work, |reg| in the
:logger construction block might need to be a proxy that collects the
services used in the block, and adds them to a dependency graph,
instead of being the registry itself (for thread-safety?)

But what about something like this happening:

reg.registerlogger) { |r| Logger.new( reg.config.filename ) }
^^^ ^^^ ^^^

In other words, there is nothing that prevents the block from referring
to registries and containers outside the block. This means that it
really can't be a proxy object--it has to be the container itself that
knows the dependency information.

Perhaps you could use a thread-local variable to keep track of the
service that is being constructed. It would need to be a stack, though,
since instantiating one service can cascade into multiple service
instantiations.

If the #register method stored a stack of service names that are
currently being constructed by each thread in a thread-local variable,
your interceptor could use that variable to relate the dependency back
to the registry...

Hmm. But this falls apart when the container itself is a dependency,
since the service that depends thus on the container could query the
container at any time, not just during the service's instantiation. This
means that the service could effectively have new dependencies at
arbitrary points during its lifecycle:

reg.registersvc) { SomeNewService.new(reg) }
reg.svc.reg.another_service
...

I guess what I'm feeling is that there is no general way to know,
definitively, all of the dependencies of a service, especially in a
dynamic language like Ruby. Without putting restrictions on when and how
services are defined (like Copland does), I'm not sure how to come up
with a general solution to this.

That said, what about a slightly less transparent approach:

reg.registerconfig) { RefreshableProxy.new { Config.new } }
reg.registerlogger) { RefreshableProxy.new( reg, :config ) {
Logger.new( reg.config.filename } }

logger = reg.logger
reg.config.refresh!
...

What the above snippet is trying to demonstrate is a kind of observer
pattern. In the first registration, we create a (imaginary)
RefreshableProxy instance that wraps the given configuration instance.
In the second, we create another RefreshableProxy instance that wraps
the Logger.new instantiation. The second one also declares itself to be
an observer of the :config service in the given container.

When #refresh! is invoked on a RefreshableProxy, it notifies all of its
observers that it changed, and the next time a method is invoked on that
proxy it re-executes its associated block. It's really a specialized
kind of deferred instantiation.

Does that make sense? It could be added to the framework so that EVERY
service is automatically wrapped in a RefreshableProxy, but that would
add a lot of overhead that would rarely be needed. Doing it manually
requires you to explicitly specify the dependency graph in the form of
observers, but also incurs a lot less overhead (in general).

Now, if that is what you were wanting, the above COULD be implemented as
a new service model:

class RefreshableServiceModel
...
end

reg.service_models[:refreshable] = RefreshableServiceModel
reg.register( :config, :model=>:refreshable ) { Config.new }
reg.register( :logger, :model=>:refreshable, bserve=>[:config] ) {
Logger.new( reg.config.filename ) }

config = reg.config
logger = reg.logger
config.refresh!
...

If you would like a concrete implementation of RefreshableServiceModel,
let me know. Otherwise, I'll leave it as an exercise for the reader.

- Jamis

 

[leon breedt]

Now, if that is what you were wanting, the above COULD be implemented as
a new service model:

Sounds workable, easier to understand as well. I don't mind having to
explicitly declare the dependencies, easier to maintain as well, no
undercover magic

The behaviour I'd want is probably just for #refresh! to cause a
#refresh! call on all the observers, but thats just implementation
details.

Thanks for the detailed responses!
Leon

 

[Jamis Buck]

Sounds workable, easier to understand as well. I don't mind having to
explicitly declare the dependencies, easier to maintain as well, no
undercover magic

The behaviour I'd want is probably just for #refresh! to cause a
#refresh! call on all the observers, but thats just implementation
details.

I actually thought of that detail after I posted the email, so I
agree--that is a feature that ought to exist in the service model as well.

Thanks for the detailed responses!

No problem! I just hope they weren't _too_ detailed. I felt like I was
rambling a bit too much... At any rate, I'm glad they were helpful.

- Jamis

 

[Eivind Eklund]

I guess what I'm feeling is that there is no general way to know,
definitively, all of the dependencies of a service, especially in a
dynamic language like Ruby. Without putting restrictions on when and how
services are defined (like Copland does), I'm not sure how to come up
with a general solution to this.

I get a feeling you're overengineering here.

Doesn't something like the following work?

class Registry
def initialize
@registry = Hash.new
@active_dependency_registrations = []
@dependencies = Hash.new
end
# This is done by method_missing in Syringe
def get(symbol)
register_dependency(symbol)
retval = nil
begin
start_dependency(symbol)
@registery[symbol].call
finally
end_dependency(symbol)
end
end
def register(symbol, &block)
@registery[symbol] = block
end
def start_dependency(symbol)
if (@active_dependencies.include? symbol)
raise "Circular dependency"
else
@active_dependencies.each do |dependent_symbol|
@dependencies[dependent_symbol] ||= []
@dependencies[dependent_symbol] |= [symbol]
end
@active_dependencies |= [symbol]
end
end
def end_dependency(symbol)
@active_dependencies -= [symbol]
end
end

It builds a dependency tree only inside the present registry.

It only count the directly called methods from call/yield, but that
should be OK for most cases.

And this is just pseudo-code - it's not the way I'd write it for
production (I'd use Set and Hash.default, and handle threads, and ...)

Eivind.

 

[Jamis Buck]

I guess what I'm feeling is that there is no general way to know,
definitively, all of the dependencies of a service, especially in a
dynamic language like Ruby. Without putting restrictions on when and how
services are defined (like Copland does), I'm not sure how to come up
with a general solution to this.

I get a feeling you're overengineering here.

Doesn't something like the following work?

class Registry
def initialize
@registry = Hash.new
@active_dependency_registrations = []
@dependencies = Hash.new
end
# This is done by method_missing in Syringe
def get(symbol)
register_dependency(symbol)
retval = nil
begin
start_dependency(symbol)
@registery[symbol].call
finally
end_dependency(symbol)
end
end
def register(symbol, &block)
@registery[symbol] = block
end
def start_dependency(symbol)
if (@active_dependencies.include? symbol)
raise "Circular dependency"
else
@active_dependencies.each do |dependent_symbol|
@dependencies[dependent_symbol] ||= []
@dependencies[dependent_symbol] |= [symbol]
end
@active_dependencies |= [symbol]
end
end
def end_dependency(symbol)
@active_dependencies -= [symbol]
end
end

My head hurts.

It builds a dependency tree only inside the present registry.

Fascinating approach, Eivind. Let me see if I read this right.

* Registering the callback is nothing special -- just storing the
callback in the hash.

* When a service is requesed, you register the requested service as a
dependency of whatever service is currently being constructed.

It only count the directly called methods from call/yield, but that
should be OK for most cases.

Indeed. However, if this kind of functionality were to be put into the
core of the framework, I would personally prefer to see it robust enough
to either capture all dependencies, regardless of where or how they are
created, or none of them. Given Syringe's current implementation, I
don't think an efficient, general-purpose solution meeting that
criterion can be devised. I'd love to be proven wrong, though.

(On a related note: I just registered a RubyForge project under the name
"Needle". I think "Needle" is easier to say, spell, and remember, and
still captures what "Syringe" was trying to imply, as well. In future
posts, I'll be referring to 'Syringe' as 'Needle', since Syringe was
just a code-name anyway.)

 

[Eivind Eklund]

Eivind Eklund wrote:

[ ... code deleted ... ]

My head hurts.

Sorry about that.

Fascinating approach, Eivind. Let me see if I read this right.

* Registering the callback is nothing special -- just storing the
callback in the hash.

* When a service is requesed, you register the requested service as a
dependency of whatever service is currently being constructed.

Exactly. And only inside the present registry.

Indeed. However, if this kind of functionality were to be put into the
core of the framework, I would personally prefer to see it robust enough
to either capture all dependencies, regardless of where or how they are
created, or none of them. Given Syringe's current implementation, I
don't think an efficient, general-purpose solution meeting that
criterion can be devised. I'd love to be proven wrong, though.

I don't think any can be devised, period. And thus, I think that the
design criterion is inappropriate - if a problem is too hard, find a
smaller and better problem to solve

In this case, I think that being able to register the dependency tree
for the cases where somebody declares all of their construction "in
the same place" might be useful. I also guess this covers at least
95% of the relevant cases, and that the cases where the holes are made
will be obvious - and that they won't be obvious if we try to engineer
a "complete" solution. And it is easy to create something that lets
us add an extra manual dependency for those cases.

Having to manually track the dependency trees separately from the
instation heavily violates the Don't Repeat Yourself principle, so if
there is any chance of this being used for complex cases, I think that
a capability to auto-record is important.

Then again, I've not used any of this in practice, so I may be totally
off the chart.

Eivind.

 

[Jamis Buck]

I don't think any can be devised, period. And thus, I think that the
design criterion is inappropriate - if a problem is too hard, find a
smaller and better problem to solve

True, but only if it's a problem worth solving. In my experience with
DI, I've never needed to query the dependency graph. That doesn't mean
it isn't a useful feature--I'm just saying it isn't something that would
get used very often.

That said, is it worth spending a ton of effort on? According to the
80-20 rule--I say not. Does the currently implementation require a
violation of DRY, as you pointed out, to accomplish dependency graph
traversal? Yes. But given how often that feature is needed, I think it's
a fair trade.

If, down the road, my assumptions are proven false, I'll be happy to rip
the internals apart and rebuild them to support this feature. But I've
got more immediately useful features to solidify first. This whole
needle framework is very experimental at this stage. If someone else
wants to hack on the sources and add dependency graph traversal, feel
free. Needle will be in CVS on RubyForge (pending approval) within the
next few days. I will be glad of any submitted patches.

In this case, I think that being able to register the dependency tree
for the cases where somebody declares all of their construction "in
the same place" might be useful. I also guess this covers at least
95% of the relevant cases, and that the cases where the holes are made
will be obvious - and that they won't be obvious if we try to engineer
a "complete" solution. And it is easy to create something that lets
us add an extra manual dependency for those cases.

True, but the holes, IMO, introduce an element of surprise. If
dependencies are tracked under 95% of the cases, and you don't
understand how the system is really working, you would end up with bugs
in your code when you declare a dependency differently from the expected
process.

I'd really rather users of needle not have to understand how needle
works its magic.

Having to manually track the dependency trees separately from the
instation heavily violates the Don't Repeat Yourself principle, so if
there is any chance of this being used for complex cases, I think that
a capability to auto-record is important.

Then again, I've not used any of this in practice, so I may be totally
off the chart.

Me, too. The only use I've seen for this is the one that Leon posted
with being able to refresh a service's implementation and have all
services that are dependant upon the refreshed service automatically
refresh as well. How often is _that_ needed? I don't know. _I've_ never
needed it, but Leon has. No one else has piped up yet to say whether
_they've_ needed it, so it's still unclear how generally useful such a
feature would be.

So, given that we're both kind of shouting into a vacuum here, we might
as well just hold off on this until more real-life use cases pop up.
I'm expecting many different iterations and rewrites of Needle before it
is really "mature", since this is an approach to DI that's never been
taken before.

Thanks, Eivind, for your feedback! It's great food for thought, and it
challenges my assumptions, which is always a good thing.