Living with a Swarm of Boids - A report from the front

[thunk]

Hi,

Haven't checked in for awhile... thanks, guys, for the info from
other threads.

The "beef" in this post, if there is any, will probably be the
"Advantages Seen" below.... as the place to start - if you think like
me.

So, where any other "flow" diagram for a project I have worked on for
(too many years > 30) would have all circles and squares, with a
database shown as a big stack of cool disks and such I have my "flow
diagram now on my wall inspiring me on... there are the usual major
classes but there is a "Swarm_Sender" rectangle, and a
"Swarm_Receiver" rectangle and a bunch of lines representing the
literally 100's (to be 1000's) of "Boids" / Agents that make the
"flight" between these two major classes.

It's been a real "trip" working with this layout day in and day and
for awhile the maintenance and additional confusion was threatening to
overwhelm me, but as usual when I keep my head down on this stuff
solutions come and if nothing else, I things like this can just "grow
on me". Now, I think, the project has grown enough, worthwhile things
are being resolved, performance has been studied and found more than
adequate (2,000 boids can be fired per second), and I have shook down
the code/refactored things a few times now and its just "holding up
nicely". I've been writing a lot of "self tests" that detect and
report a lot of basic stuff before anything even starts... all based
on a lot of "reflection". Turns out, I think, that this kind of
system work is more critical than usual when working with these
things.

A FEW NOTES ON THE "LIFECYCLE" OF A BOID

*Every boid is a single text file, which is a proper Class written
with the key method "written" in (generally) 4 sections supported by 4
DSL_ classes.

*Phase one of the program is a bunch of checks for data module
integrity and such. The Boids are loaded from designated subdir(s)
and sequenced and "readied for flight" so to say. (There are multiple
flights...)

*Phase two 'fires!' the Boids sequentially
Each Boid is executed/supported by a sequence of DSL classes (one
each section)
Each Boid "rendezvous"/docks with a Helper Class where the
heavy lifting calcs/logic is done.
Each Boid makes assertions per the authors choices - and
records the results are recorded in gory detail into a record
the the Boid "births" - (rather like a black box for an
airplane?)
Each Boid is "Debriefed" in a "overview sort of way" - which is
immediately reflected into a "dashboard" like thingy I have created
(but which is disconnected for now to avoid graphics)
The "black box" records are tucked away for debugging, or really geeky
clients to examine via the "dashboard" features.



ADVANTAGES SEEN TO THIS DESIGN:

ONE
The normal curve for software complexity versus task complexity would
probably show a increasingly upward sloping curve for the software,
right?

Well, with this design that is probably still true, and it is steeper
at the gitgo - that is for sure- BUT, and I believe it is starting
to be born out that after a certain complexity has been handled (THANK
YOU RUBY!!!) the curve is considerably less steep. Seems like a rpm
"shifting pattern" kinda thing going on here.... (bet I'm in 3rd
now).

Is there a "special" exception to handle? This is the payload of this
design technique - you write a Boid or adopt and existing one. NOTHING
else has to change (!) (THANK YOU RUBY!!! )

Entire "alternate views" could be forged by controlling the name of a
subdir assuming the format of the data can be shared.

In our domain one Clinic or Hospital might keep a "local layer" on top
of a national one. These issues become "subdir" name level stuff.

(Firing Boid Swarms from multiple views could be fascinating too!)

TWO
The Boids have been kept really simple. Amazingly simple, I think.
The guts of the things are the "Assertion" statements and really, this
has been reduced to "building a natural language sentence" from 2 or 3
"picklists". The "messages" attached to the assertions are the most
time consuming / require the most time and effort - and they are only
really needed for examining the logic / attribs gathered to make the
decisions. I had this all working on-line.

Because the Boids can be authored on-line, and the nature of the Web,
an expert in South Africa could document a problem (perhaps using a
even simpler Boid type - the Review type - makes no assertions) and it
would show up in Boston minutes later. Mind boggling stuff?


SUMMARY

Nothing is done until it is done, it is clear to me that what I am
doing so far with only about 100 Boids could have been done much
faster and simpler using any one of a number of programming
techniques.
It would also execute much faster if done in the good old 'c'
language.

However, I believe that it is not only a fascinating design objective,
but also some large commercial potential to opening up the 'rules' of
the system for outsiders to contribute to via the web.

Where timeliness is of value, complexity is significant/or subject to
(rapid)change, and a passive "broadcast" type system fits the use
patterns this technique seems to fit very well.


Sincerely,

Thunk


PS somebody on this forum discouraged me in a phone call on this
project, saying "that it would be 'high maintenance'" and I knew he
was right, but I think I'm going to be proven right also, and that the
extra effort is going to be worth it.

 

[thunk]

I did a little arithmetic after sending my note off early this AM and
it seems that with only about 120 Boids in near future (and thousands
expected) there is about an approximate 50:50 balance between the
number of 'lines' (of code) inside the Boids total and total of the
"support system" for them including the classes which represent mostly
data. About 8,000 each (lots of white space & comments but somewhat
the same throughout). What is interesting to me about this is that
even if 3/4 of the code from the 'Boids" is elminated as being only
necessary because of the design, there is still a nice "balance"
achieved at about 1000 Boids.

Given that some of my work to develop a graphical "Dashboard", given
that I have now dealt with the details of 3 different domains, and
given that I'm using a lot of simulation in this project I have rather
acquired the overall impression that a system first becomes useful/
mature at somewhere over 1000 boids. The probability of getting a
useful hit or two on a less than 1000 Boid system seems about the same
as asking a WalMart clerk a technical question

I also just did some checking on the rather latest and greatest
computer speeds and it seems my 3 year old system (a proud Hackintosh
that has never seen OSX) is now running at about 1/2 the speed of the
most fasterestish desktop. This, I think, means that given the total
resources of a system that some 4,000 Boids can "fire!" per second.
(Sorry any Ruby purists, but I stuck a exclamation mark on my first
"fire" method about a year ago and I won't take it away

Anyway, in something like 2 seconds a state-of-the-artish micro can
mechanically throw a set of 1,000 "Boids" at a single question. Now,
even at my primitive state of development on this less than 10% of the
Boids are actually fired!, even a rather broad set "questions'. Where
I'm going with this is that a 10,000 Boid system, which would
represent a pretty good set of knowledge for what I'm seeing - can
effectively fired! in less than 3 seconds.

I would love to hook a active forum up to one of these things
somehow.

So, will my 1992 Model 7020 Huffy Seatpost 'fit' in a 1958 Flying
Pigeon (based on a 1942 Raleigh)?

Well, that's a million Boid question, and I'm tired....


Thunk

 

[thunk]

Whoops!

Anyway, in something like 2 seconds a state-of-the-artish micro can
mechanically throw a set of 1,000 "Boids" at a single question.

Should have been:

Anyway, in something like 2 seconds a state-of-the-artish micro can
mechanically throw a set of 8,000 "Boids" at a problem set.


Sorry

 

[Robert Klemme]

However, I believe that it is not only a fascinating design objective,
but also some large commercial potential to opening up the 'rules' of
the system for outsiders to contribute to via the web.

Actually from what you've written your design (or the core idea behind
the design) hasn't become clear to me. So far I see a framework that
reads program code from particular file system locations and executes it
according to rules imposed by the framework. Until now I fail to
recognize the revolutionary novelty in this that you seem to claim for
this design. Also, the aim of the exercise is not clear to me.
Initially I thought you were building software to handle large flow
diagrams; then it sounded like you actually want to simulate or execute
those flows...

Kind regards

robert

 

[thunk]

Robert,

Whatever else this project is, I think it has to be a rather "extreme
design". Will you give me that? Development and debugging has been
slow and involved early on. Only after multiple iterations have I
been able to simplified and the roles clarified to where things are
now. The set of "Boids" have determined the names and functions of
most of the other project parts. There's a strong mental model to
adjust to, but some elements of the project like the need for a
"categorization" scheme did not become evident until I was thinking,
"ok I have the results from 100 fired! boids, what can I do with
them? Early on I had developed the concept of a "result record" and
the result record could be used by subsequent passes to build reports
- I spent a week testing it back when. What I didn't see at first was
the use of "output panel" that acts on the results of the Boids like a
big billboard passively announcing the number of various "hits" in
various subcategories. It all makes sense, and seems totally natural
when you see it in the context of "your domain". I really don't think
it was so obvious when I was designing the DSLs the support the
various sections of the Boids.

In he current design there is a "transmitter object", and a "receiver
Object". The receiver object "debriefs" the Boids and send info to
the "Control Panel" and into open "Work Sheets" (probably particular
to my domain). The Boid author specifies the assertions he needs to
arrive at a passed/failed conclusion. These things build up into
something pretty interesting with depth. Each Boid, so far, is
limited to using a single helper_class - where the "real" logic and
calculations are concentrated. The boid is a sort of "gatherer shell"
or something - but that's the whole idea - they must be kept simple so
that non-programmer types can design them.

What's exciting about this? Hmmm. What's exciting to me is that the
system could grow from outside experts from a network of say 1,000 to
10,000 boids "organically" without the need to change one line of code
in the framework. What I think should be 'exciting' to
entrepreneurial types is that this can be done using outside resources
- like a forum. So the whole thing becomes something like a "knowledge
farm". I'm not sure where the "magic" is, its all "just ruby" but I'm
watching the output of some pretty involved problem solving getting
done quickly - and I'm no where near the 1000 Boid "tipping point"
that I consider to be the design break-even point for this system
versus a strictly "conventional design".

Then, if I'm right about the run-up from 1000, to 10,000Boids this
will be getting to pretty interesting - at least to me.

Thunk

PS There is this inherent power to the web that allows a contributor
to be anybody / anyplace / anytime. And there is this ability to
process 10,000 of these "logic capsules"/Boids or whatever in about 2
seconds. And there is "time value" attached to knowing something "out
there" changed that effects something I'm about to do. This all fits
together somehow, "my domain" gets close enough to justify my efforts
to me, I think there are domains where the power of this design could
be leveraged much farther - I keep thinking of medical procedures
where lives are on the line and changes are happening everyday -
around the world. A 100,000Boid system could be fired at the records
of every patient every day for about the same resources (per hospital)
as a couple hours of video game playing. Nothing like this seems to
be happening. Why?

 

[Philip Rhoades]

Thunk,

I have been reading the posts but I still have problems understanding
what this setup is actually for - I have an interest in Ruby and
biological simulations so the word "swarm" captured my attention. Can
you describe what you are attempting to do in a more simple/overview way
so I can see if I should be interested in this or not?

Thanks!,

Phil.

Robert,

Whatever else this project is, I think it has to be a rather "extreme
design". Will you give me that? Development and debugging has been
slow and involved early on. Only after multiple iterations have I
been able to simplified and the roles clarified to where things are
now. The set of "Boids" have determined the names and functions of
most of the other project parts. There's a strong mental model to
adjust to, but some elements of the project like the need for a
"categorization" scheme did not become evident until I was thinking,
"ok I have the results from 100 fired! boids, what can I do with
them? Early on I had developed the concept of a "result record" and
the result record could be used by subsequent passes to build reports
- I spent a week testing it back when. What I didn't see at first was
the use of "output panel" that acts on the results of the Boids like a
big billboard passively announcing the number of various "hits" in
various subcategories. It all makes sense, and seems totally natural
when you see it in the context of "your domain". I really don't think
it was so obvious when I was designing the DSLs the support the
various sections of the Boids.

In he current design there is a "transmitter object", and a "receiver
Object". The receiver object "debriefs" the Boids and send info to
the "Control Panel" and into open "Work Sheets" (probably particular
to my domain). The Boid author specifies the assertions he needs to
arrive at a passed/failed conclusion. These things build up into
something pretty interesting with depth. Each Boid, so far, is
limited to using a single helper_class - where the "real" logic and
calculations are concentrated. The boid is a sort of "gatherer shell"
or something - but that's the whole idea - they must be kept simple so
that non-programmer types can design them.

What's exciting about this? Hmmm. What's exciting to me is that the
system could grow from outside experts from a network of say 1,000 to
10,000 boids "organically" without the need to change one line of code
in the framework. What I think should be 'exciting' to
entrepreneurial types is that this can be done using outside resources
- like a forum. So the whole thing becomes something like a "knowledge
farm". I'm not sure where the "magic" is, its all "just ruby" but I'm
watching the output of some pretty involved problem solving getting
done quickly - and I'm no where near the 1000 Boid "tipping point"
that I consider to be the design break-even point for this system
versus a strictly "conventional design".

Then, if I'm right about the run-up from 1000, to 10,000Boids this
will be getting to pretty interesting - at least to me.

Thunk

PS There is this inherent power to the web that allows a contributor
to be anybody / anyplace / anytime. And there is this ability to
process 10,000 of these "logic capsules"/Boids or whatever in about 2
seconds. And there is "time value" attached to knowing something "out
there" changed that effects something I'm about to do. This all fits
together somehow, "my domain" gets close enough to justify my efforts
to me, I think there are domains where the power of this design could
be leveraged much farther - I keep thinking of medical procedures
where lives are on the line and changes are happening everyday -
around the world. A 100,000Boid system could be fired at the records
of every patient every day for about the same resources (per hospital)
as a couple hours of video game playing. Nothing like this seems to
be happening. Why?

--
Philip Rhoades

GPO Box 3411
Sydney NSW 2001
Australia
E-mail: (e-mail address removed)

 

[thunk]

Thunk,

I have been reading the posts but I still have problems understanding
what this setup is actually for - I have an interest in Ruby and
biological simulations so the word "swarm" captured my attention.  Can
you describe what you are attempting to do in a more simple/overview way
so I can see if I should be interested in this or not?

Thanks!,

Phil.







--
Philip Rhoades

GPO Box 3411
Sydney NSW      2001
Australia
E-mail:  (e-mail address removed)

Let me know if the Vocabulary effort helped? It's rather "extreme"
and frankly makes no sense at all performance or development effort on
a "from scratch basis" unless the app has some real complexity or the
externally contributed aspect can grow the knowledge-base. I think it
is ideal to get hooked up to forums as a sort of parallel to the text
messages but building up the knowledge-base. I have concocted a
couple examples from the medical field - it starts jumping out at you
when you see it. And, its fun

 

[Ryan Davis]

Whatever else this project is, I think it has to be a rather "extreme
design". Will you give me that?

How can we give you that? You're rambling here on and on about something =
you've already said we'll never see.

 

[thunk]

How can we give you that? You're rambling here on and on about something you've already said we'll never see.

Oh, you're probably right, sharing actual ruby experience and concepts
has nothing to do with a ruby discussion group.

 

[Philip Rhoades]

Thunk,

Let me know if the Vocabulary effort helped?

Not really . .

It's rather "extreme"
and frankly makes no sense at all performance or development effort on
a "from scratch basis" unless the app has some real complexity or the
externally contributed aspect can grow the knowledge-base. I think it
is ideal to get hooked up to forums as a sort of parallel to the text
messages but building up the knowledge-base. I have concocted a
couple examples from the medical field - it starts jumping out at you
when you see it. And, its fun

Fun maybe but I still don't understand - I think we need a concise,
straightforward summary. Where can I see examples or diagrams?

When I think of "swarms" I think of:

http://www.swarm.org/index.php/Main_Page

- is what you are proposing an agent or individual-based model that
could be used for biological populations?

Thanks,

Phil.
--
Philip Rhoades

GPO Box 3411
Sydney NSW 2001
Australia
E-mail: (e-mail address removed)

 

[Ehsanul Hoque]

g you've already said we'll never see.

=20
=20
Oh=2C you're probably right=2C sharing actual ruby experience and concept= s
has nothing to do with a ruby discussion group.

Show me the moneyy - uh - I mean the code.
Really=2C you make close to no sense from what I've read=2C and I think oth=
ers here are as confused as I am. If you could show what you mean in code (=
while explaining it in direct and concise terms if at all possible)=2C I'm =
sure people will appreciate your creation a little more=2C or at the very l=
east be able to understand it.
- Ehsanul =20
_________________________________________________________________
Hotmail: Trusted email with powerful SPAM protection.
http://clk.atdmt.com/GBL/go/210850553/direct/01/=

 

[Dido Sevilla]

Note that you haven't even bothered to define what the heck a 'boid'
is from your so-called 'system vocabulary'. You say this:

BOID
Several types - makes assertions or can be a review, or can
contribute to reports.
Several SECTIONS- Headings, Attributes, Assertions,Messages,
Conclusion each a DSL. More coming.
ScrptRec - sort of the "black box" a recording of everything and
anything that happens to the Boid that can be of interest - names of
author, intermediate calculation results, everything gets recorded
into this object. Nothing else is left when the Boid is finished.

Which tells me and everyone else who has no clue about what you're
talking about absolutely nothing about what you're trying to do or
what is motivating your exposition. This is not a definition. A
definition tells people what something is, this just says what it
might be able to do, apparently without even considering that key
question. Remember that no one else here knows what it is you're
trying to do, and from the way you've gone about things we seem to
know even less than ever.

--=20
=E6=99=AE=E9=80=9A=E3=81=98=E3=82=83=E3=81=AA=E3=81=84=E3=81=AE=E3=81=8C=E5=
=BD=93=E7=84=B6=E3=81=AA=E3=82=89=E7=AD=94=E3=81=88=E3=82=8B=E7=A7=81=E3=81=
=AF=E4=BD=95=E3=81=8C=E3=81=A7=E3=81=8D=E3=82=8B=EF=BC=9F
=E6=99=AE=E9=80=9A=E3=81=A7=E3=82=82=E6=99=AE=E9=80=9A=E3=81=98=E3=82=83=E3=
=81=AA=E3=81=8F=E3=81=A6=E6=84=9F=E3=81=98=E3=82=8B=E3=81=BE=E3=81=BE=E6=84=
=9F=E3=81=98=E3=82=8B=E3=81=93=E3=81=A8=E3=81=A0=E3=81=91=E3=82=92=E3=81=99=
=E3=82=8B=E3=82=88=EF=BC=81
http://stormwyrm.blogspot.com

 

[Ryan Davis]

something you've already said we'll never see.

=20
Oh, you're probably right, sharing actual ruby experience and concepts
has nothing to do with a ruby discussion group.

Look at what you originally said and look at my response. You haven't =
shown us ANYTHING that implies you actually have an "extreme" design (or =
ANY design for that matter).

Further, you're not actually sharing ruby experience or concepts. You =
haven't said a damn thing about ruby at all nor have you said anything =
about your design. You've said a lot, but to us there is no signal, just =
noise. Like Ehsanul said, show us the money / code. I'm genuinely =
curious about what you're working on, but the more you post noise, the =
less curious I get.

 

[Robert Klemme]

Whatever else this project is, I think it has to be a rather "extreme
design". Will you give me that?

Since I still haven't grasped what your project is all about I can't
make any such judgments. Maybe it's just me but I am starting to
suspect that the lack of clear explanation that I am perceiving hint's
at a lack of understanding on your part. Usually people who have a
clear understanding of a matter can explain it in a way that others
can pick up the concept. The fact that several people do not
understand what all this is about might just show that we are thinking
along totally different lines than you. Either way, since you are
posting here I assume you want to enter a dialogue and get meaningful
responses. It will help you to reach that end to provide an
explanation that actually is understood by your peers.

Cheers

robert

 

[Aldric Giacomoni]

Since I still haven't grasped what your project is all about I can't
make any such judgments. Maybe it's just me but I am starting to
suspect that the lack of clear explanation that I am perceiving hint's
at a lack of understanding on your part. Usually people who have a
clear understanding of a matter can explain it in a way that others
can pick up the concept.

http://en.wikipedia.org/wiki/Boids ?
I remember him posting about this a while back, but that's all. Maybe
he's building a new Rails, but instead of being a web development
framework, it's for Boids? Ooh, that'd be fancy.