Real-time image processing in Ruby

[John Koschwanez]

I'm a Ruby newbie - "Programming Ruby" was great Xmas break reading!
I write image analysis and processing code to analyze single cells under
a microscope in real-time and to automate lab equipment. I now write
solely in C++ for PC, but I would love to use Ruby to quickly try new
ideas without plowing through C++ code. I have read through the forums
and I see Ara Howard is using C extensions to process images - I'll try
the same thing. I have some questions for the group:
- I capture images from a camera, process and analyze them, and display
them at ~10 fps. Ruby-v4l (capture) is for Linux only. Tk is probably
too slow, and someone recommended using OpenGL for display. Has anyone
had success with image capture or display using Ruby? What worked? Any
advice?
- Is anyone using pure Ruby to process images real-time? RMagick is too
limited for my applications. I doubt Ruby would match the speed of C,
but it might be fun to try.
Thanks, John

 

[Ron M]

Correct me if I'm wrong, but image processing is one of those things
that plays directly into the weaknesses of the current Ruby
implementation.

I disagree. Image processing is an area where even in Java/C#/C/etc
you're probably better off using highly tuned matrix libraries than
rewriting such basic operations from scratch; and Ruby has a pretty
good matrix library (NArray).

And under the assumption that regardless of the language most of
the CPU time will be used inside such a matrix library, the
overhead of the language wrapping it shouldn't matter much.



If you do find yourself writing ruby loops to touch large sets
of individual pixels, however, you should probably consider
extending NArray in C instead - hopefully with some pretty
general purpose function so many ruby algorithms could make
use of the same extensions. (for example, I'm not sure if
NArray has a nice way of specifying a mask to only process
part of an image).

- R

 

[Phil Tomson]

I'm a Ruby newbie - "Programming Ruby" was great Xmas break reading!
I write image analysis and processing code to analyze single cells under
a microscope in real-time and to automate lab equipment. I now write
solely in C++ for PC, but I would love to use Ruby to quickly try new
ideas without plowing through C++ code. I have read through the forums
and I see Ara Howard is using C extensions to process images - I'll try
the same thing. I have some questions for the group:
- I capture images from a camera, process and analyze them, and display
them at ~10 fps. Ruby-v4l (capture) is for Linux only. Tk is probably
too slow, and someone recommended using OpenGL for display. Has anyone
had success with image capture or display using Ruby? What worked? Any
advice?
- Is anyone using pure Ruby to process images real-time? RMagick is too
limited for my applications. I doubt Ruby would match the speed of C,
but it might be fun to try.

Ruby will probably be too slow for your needs (I doubt you'll be able to
make even 10fps unless you write a lot of your code in a C extension). What
if you take a C or C++ image processing lib (like CImg) and wrap it for use
from Ruby? Well, CImg might be a tough one as it relies (too much) on macros
- that could make it tough to wrap, I suspect.

Another option might be to use Ruby as a code generator. I was working on a
project last summer where we were using a Matrox card for image
capture/processing. We
were using the Matrox image processing API which is a bit of a pain to work
with (you end up having to violate DRY all the time the way it's set up). So
what I started doing was to create a DSL (domain specific langauge) which
could generate the C code using the Matrox image processing API. I didn't
finish it, but I think the appraoch had potential. Basically, you could write
a bit of Ruby code that would then generate a whole lot more C code (something
like 4 to 8X the number of lines; and you didnt' need to violate the DRY
principle). So the description of what you wanted to do was in fairly
compact, easier to understand Ruby code which generated the C code and
compiled it. While Ruby itself isn't fast enough to do realtime image
processing, you really only need to describe how you want the image processed
and then generate C code which runs many, many times faster. And you wouldn't
even need to write any C code (after you've worked out the code generation
details, that is).

I think image processing is very amenable to this sort of highlevel
description, because most image processing is just a matter of a series of
transfromations:

Captured Image -> transform A -> transform B ... transform N -> output image.

In our case what we needed was to be able to easily plug in different
transforms to find out which one(s) would give us the most desireable results.
Perhaps in your case you already know what transforms you need and what order
they should be in, if that's the case you may not find much benefit from this
approach.

Phil

 

[M. Edward (Ed) Borasky]

Have a look at the Perl Data Language (PDL). This was designed to do
high-performance image processing with a Perl user interface. I suspect
the underlying C code could be interfaced to Ruby as easily as it was to
Perl. If you're going to do serious number crunching of any kind in
Ruby, you probably want to use C code to do the work.

 

[ara.t.howard]

I'm a Ruby newbie - "Programming Ruby" was great Xmas break reading!
I write image analysis and processing code to analyze single cells under
a microscope in real-time and to automate lab equipment. I now write
solely in C++ for PC, but I would love to use Ruby to quickly try new
ideas without plowing through C++ code. I have read through the forums
and I see Ara Howard is using C extensions to process images - I'll try
the same thing. I have some questions for the group:
- I capture images from a camera, process and analyze them, and display
them at ~10 fps. Ruby-v4l (capture) is for Linux only. Tk is probably
too slow, and someone recommended using OpenGL for display. Has anyone
had success with image capture or display using Ruby? What worked? Any
advice?
- Is anyone using pure Ruby to process images real-time? RMagick is too
limited for my applications. I doubt Ruby would match the speed of C,
but it might be fun to try.
Thanks, John

i'm not doing anything near-realtime, but you may want to check this out:

http://artengine.ca/gridflow/

my understanding is that it's used for realtime video processing, so images
might be possible too.

i must say, however, that it seems like doing what you are doing, especially
in linux only, might be possible with rmagick only - just dump the frames in
/dev/shm and go...

another option, quite suitable for prototyping only, is just using narray : i
use it together with a hack that allows the grids to be mmap'd quantities so
'image processing' is nothing but matrix ops... narray ships with a simple
method of displaying images under x. at 10 fps this might just hack it.

regards.


-a
--
===============================================================================
| ara [dot] t [dot] howard [at] noaa [dot] gov
| all happiness comes from the desire for others to be happy. all misery
| comes from the desire for oneself to be happy.
| -- bodhicaryavatara
===============================================================================

 

[ara.t.howard]

I disagree. Image processing is an area where even in Java/C#/C/etc
you're probably better off using highly tuned matrix libraries than
rewriting such basic operations from scratch; and Ruby has a pretty
good matrix library (NArray).

And under the assumption that regardless of the language most of
the CPU time will be used inside such a matrix library, the
overhead of the language wrapping it shouldn't matter much.

absolutely. unless you start looping... this is the tradeoff for high-level
languages.

If you do find yourself writing ruby loops to touch large sets
of individual pixels, however, you should probably consider
extending NArray in C instead - hopefully with some pretty
general purpose function so many ruby algorithms could make
use of the same extensions. (for example, I'm not sure if
NArray has a nice way of specifying a mask to only process
part of an image).

and a combination of NArray and RubyInline and/or ruby dl would make for very
rapid prototyping indeed.

-a
--
===============================================================================
| ara [dot] t [dot] howard [at] noaa [dot] gov
| all happiness comes from the desire for others to be happy. all misery
| comes from the desire for oneself to be happy.
| -- bodhicaryavatara
===============================================================================

 

[ara.t.howard]

- Is anyone using pure Ruby to process images real-time? RMagick is too
limited for my applications.

i missed this the first time through...

this is totally uncharted territory, but see

http://raa.ruby-lang.org/project/ruby-vtk/
http://public.kitware.com/VTK/

i'm guessing you need something like that if rmagick is too limited... vtk
certainly has tons of algorithms... what beast to compile though!

cheers.

-a
--
===============================================================================
| ara [dot] t [dot] howard [at] noaa [dot] gov
| all happiness comes from the desire for others to be happy. all misery
| comes from the desire for oneself to be happy.
| -- bodhicaryavatara
===============================================================================

 

[John Koschwanez]

and a combination of NArray and RubyInline and/or ruby dl would make for
very
rapid prototyping indeed.

-a

Thanks - I'll try NArray with RubyInline. I was about to complain that
something so useful is not included in the stdlib, but I then saw the
recent thread covering that subject.
Someday...RubyLab!

 

[Detlef Reichl]

Am Donnerstag, den 29.12.2005, 08:08 +0900 schrieb John Koschwanez:

- I capture images from a camera, process and analyze them, and display
them at ~10 fps. Ruby-v4l (capture) is for Linux only. Tk is probably
too slow, and someone recommended using OpenGL for display. Has anyone
had success with image capture or display using Ruby? What worked? Any
advice?

Hi,

cause ruby-v4l was to limited for me i've started to wirte Wc, a webcam
library (I think you can use it also with other camseras, but i didn't
tested it). For now i've tested it with six different cams (and kernel
modules) and v4l (version 1). At this moment i'm on add 4vl2 support and
will release a preview in the next days. If *bsd has something like v4l
(i don't know), suport for this will also be added some day... Support
for windoze will not come.

Simple image prossesing linke rotation, fliping, croping, mixing, ... is
provided. Nearly the complete library is written in C and to add other
functions is very easy.

For displaying i use the ruby bindings of Gtk+. The overhead of the
grapics library for the output is very low. On my 1.23 MHz Mac Mini a
viewer at ~10 fps generates ~20-40% system load. Higher frame rates are
not possible for me, cause i have connected several cams at once and
they have to share the bandwidth (and it is not possible to use USB2 on
an external hub, due of the limitions of the usb-implimentation of
linux).

You can find it at: http://dr.dhis.org/webcam/index.rbx
With the package you will get some examples like the camera viewer.

i'm open for sugestions to extend the library.

Cheers
detlef