motion-detection using an optical flow-field

Over the past couple of weeks I’ve been working on a fun new project with a few friends. Not going to give away too much yet, but part of the project involved setting up a webcam pointing out the window of our apartment to take photos of people as they walk by on the street… see?

Since we want to only take photos of people from the front (walking right to left), one of the problems that I ran into while working on the motion-detection script in Processing is how to get a good idea of the direction of motion in the frame.

Originally I created a little class that allows you to draw boxes on the screen in order to detect a change in the average brightness of the pixels in the target areas. This is what it looked like:

By checking to see the order in which the two areas were “hit” I could tell what direction the person was moving in. This worked pretty well for a while, but I quickly ran into a problem when there were a group of people walking in the wrong direction:

When there were multiple people in the frame, I would still be hitting both sensors and generating a lot of “false positives” (this street gets a lot of traffic). I knew that I had to find a better way to sense the direction of motion. A little digging around led me to this rad Optical Flow Processing sketch created by Hidetoshi Shimodaira (video). Here he has derived the motion in the video and visualizes it with a grid of colorful vectors. The color and direction of the lines signifies the motion in the frame. Blue = left, red = right, purple = up, and yellow = down.

So, I ported this code over into a class. By finding the sum of all the vectors’ horizontal forces, this class lets me check the overall direction of motion in the frame at any given point in time. If the sum is positive, I know that the motion is going left to right. Negative means it’s right to left. And the resulting captures are way more accurate:

Besides the practical value, I also just really love the aesthetic of these colorful flow field vectors. This one in particular is a favorite. You see only a foot in the bottom-left of the frame, but the flow field trail gives you a sense for the energy of that person who just jumped through the frame, like motion lines in a comic book or cartoon.

And another fun effect results from having multiple bodies moving in opposite directions in the frame. I love how here you can see the opposing forces crashing into each other:

The OpticalFlow class I created from Hidetoshi Shimodaira’s sketch can be found on GitHub HERE.