Before Microsoft's Kinnect...

there was optical flow

The idea of having two images and calculate the speed and direction of an object has fascinated most computer vision researchers in the last two decades. According to Wikipedia, optical flow is the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer (an eye or a camera) and the scene.

For a more complete terminology check here (but try to exclude the maths).
The basic problem is: Establish the correspondence of multiple images to each other, pixel by pixel.
There are many ways to do it, but as someone can imagine performance was the show stopper most of the times. Well, until recently.

Lots and lots algorithms were written and optimized to help us solve this difficult problem. I guess the results (that are both accurate and in real-time) led Microsoft to release the Kinnect device, which is actually an all-in-one-dream-comes-true-device. Diging into history, a small paper by a russian student shows the root of the problem and all the people who worked with it.
What I take as a starting point is the publication of Horn and Schunck method "Determing Optical Flow" back in 1980. Later on, P.K. Horn would publish a book with a set of algorithms under the title "Machine Vision", which I guess started the whole scientific wave of computer vision, which is still more than active nowadays (saddly, thanks to military projects).

So, why is so interesting about it?
The reason is simple. It's the basis for the most high-tech software industries:

     - Image & video Compression
     - Object detection & tracking
     - Pattern extraction
     - Robotics

But apart from any scientific interest optical flow is impressive! Impressive for everyone. That's why it drew the attention of many film makers, designers and programmers to implement their creative ideas and create a whole new wave in the computer history known as Modern Visual Arts (or what I like to call it Computer Aided Interactive Visual Arts - CAIVA). And the wave is about to become a trend. Not only experimentally but comercially too (e.g. latest Nokia Lumia events). Pioneer in this field? Of course Vincent!

So, roughly someone can argue that optical flor is the basis for:

     - the majority of special holywood-like effects (it all began with "The Matrix")
     - interactive installations
     - video surveilance and many more.

So, let's stick to the "impressive" part of the subject. I won't try to explain the algorithms as they contain enough maths to scare most of the people. Just to mention the problem itself is by it's nature unsolved (mathematically speaking). You need to introduce a set of constraints (phase, smoothness, pixel locality etc) in order to eliminate the results. My initial pursuit on the field dates back to 2007. The outcome of my investigation is a little complex that it would better to explain it in parts:

  • Initially I wanted to implemented the so-famous Horn and Schunck method as part of a university assignment.
  • Then I thought to compare it with other methods. I picked Lucas & Kanade and Nagel ones...
  • Hey, numbers are for mathematicians lets visualize a little bit the results with various methods...
  • Now we're talking... Let's fine tweak the parameters. Convert them to dynamic methods and add some configuration...
  • Then I realised that image processing is fun. I added a bunch of filters (blur, edge extraction, dust removal etc) to tweak even more the flow. Make all these dynamic too...
  • What's better than an image? Image sequences! Added GIF support...
  • What's better than an image sequence? Video! Added video support...
  • Ok, sounds like a lot of stuff... Let's bunch all these things into a GUI with options and dialogs and everything...
  • Good. Let's add a few things on top to make it a little more artsy fartsy (mail support, PVR, timings etc)...
  • Ready to go. Need a name! Damn! Ok, lets say Optical Flow Inquiring Study (OFIS, greek for snake) desing a cool logo (thanks George) and done.

So this study ended up with a single executable written in C# based on .NET 2.0. The only drwaback I can think of is that it supports only inputs of a max 300 x 240 pixels analysis (excluding video input - it's automatically downscaled in this case). To increase compatibility with Windows 7 (originally it was targeted only for Windows XP), I recompiled it using .NET 4.5 and MS Visual Studio 2010. Due to this "upgrade", you may notice some GUI related defects but functionality is smooth.


Please free to download and give it try. Unfortunately it doesn't come with any manual unless you're greek. But I will update if I find anything relevant in english. I believe this program can give you a more than enough insight on the whole optical flow study field. Some sample data (both pictures and animated GIF) are included to give it a try.

DOWNLOAD

Not convinced yet? Check this small video demonstrating few of the program's capabilitites in real time...

If you are concerned on the performance, please bear in mind that the above sequence was made using:

  • A crappy Centrino laptop, which was capturing video at the same time of processing
  • A cheap webcam

And here is a high level structure of the program's "core":

I am willing to provide the source code in case anyone is interested in it. Just contact me!

Used tags: ,

No comments

(optional field)
(optional field)
Remember personal info?
Small print: All html tags except <b> and <i> will be removed from your comment. You can make links by just typing the url or mail-address.