Stereo Vision with Kinect (xbox360 device) on i.MX6 processor

After a long time with no posts, here it goes one more !

It is known on Image processing based projects that works as: classifiers, object tracker, and etc., that the a bad segmentation procedure could fail the entire project.

Based on the gesture recognition project already presented on the lasts posts, we can clearly see if we want to the application works fine, with every person, independent of skin color, then the user needs to wear a glove or another accessory that will differentiate from the background, also, in a gesture recognition demo, only movements on X and Y axis is detected efficiently.

It is because in images provided by a single camera, we need to do segmentation based on colors and then check the contours for the desired pattern. If the background has a color most likely with the skin color or the glove color, then we have a serious problem for the segmentation par. A lot of noise will appear on the final image and the classification/recognition will not work properly.

Unless you are working on a project that will classify/track/detect some object in a controlled environment, one camera is sufficient, but if you are aiming a project that could work on every place, so we need a different approach.

The solution for it is quite simple, we just need to add another camera !

With 2 cameras and the correct setup (alignment and distance between them), we have 2 images with different perspectives from the same point, it is, a 3D image.

There are some algorithms on the web that correlates the images from the 2 cameras and returns the disparity map, that is, the depth image.

With a depth image, the color segmentation on a project can be avoided and a segmentation based on distance can be used instead, and also, gestures more complex could be recognized.

Unfortuntely, the disparty map calculation is a heavy to the cpu do it alone, so we have the following options to improve the performance:

1 - use OpenCL (share the tasks with the GPU)
2 - use a device that already release depth images from 2 different sensors.

As we are working on a embedded system, the OpenCL port for a embedded system is not so complete, so we could face some issues and spent a lot of time on it.

On the other hand we could use a device which gives us the desired depth image. The chosen device for this task was the Microsoft Kinect. Yes ! that one used for gaming on Xbox360. Of course there is another device you could use, for example: http://www.tyzx.com/products/DeepSeaG3.html

As I already have a kinect to test (only for work, because for playing I still prefer my PS3 heheheheheh), as a demo we are going to use the kinect.

The library used for this project was the OpenKinect and you can get more information here: www.openkinect.org

Since I use Linux on my projects (LTIB), I´m going to show how to build the OpenKinect library for this platform:


NOTE: As the i.MX6 is supported on the i.MX53 BSP.

1 - Freescale´s linux BSP: https://www.freescale.com/webapp/Download?colCode=L2.6.35_11.03_ER_SOURCE&prodCode=LEIMX&appType=license&location=null&fpsp=1&Parent_nodeId=1276810298241720831102&Parent_pageType=product

2 - follow the documentation on how to install the ltib on your host linux

3 - once you have the linux running on your board we can now start building the openkinect driver.

4 - clone the openkinect git to your host: git clone git://github.com/OpenKinect/libfreenect.git

5 - it will create a freenect folder, cd to it: cd freenect

6 - create a build folder: mk build

7 - get into the ltib shell using the ltib -m command: home/ltib> ./ltib -m shell

8 - as openkinect is intended for x86, the sample apps are based on OpenGL with GLUT, so we need to avoid building any sample code, just edit the file: ../freenect/CMakeLists.txt and at line 51~63 turn the build options to off.


9 - once in the ltib shell (LTIB>) go to the build folder and type: cmake ..

10 - if every thing goes fine the log output should look like:

LTIB> cmake ..
-- The C compiler identification is GNU
-- The CXX compiler identification is GNU
-- Check for working C compiler: /opt/freescale/ltib/usr/spoof/gcc
-- Check for working C compiler: /opt/freescale/ltib/usr/spoof/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /opt/freescale/ltib/usr/spoof/g++
-- Check for working CXX compiler: /opt/freescale/ltib/usr/spoof/g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Operating system is Linux
-- Got System Processor i686
-- libfreenect will be installed to /usr/local
-- Headers will be installed to /usr/local/include/libfreenect
-- Libraries will be installed to /usr/local/lib
-- Found libusb-1.0:
-- - Includes: /usr/include
-- - Libraries: /usr/lib/libusb-1.0.so
-- Check if the system is big endian
-- Searching 16 bit integer
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of unsigned short
-- Check size of unsigned short - done
-- Using unsigned short
-- Check if the system is big endian - little endian
-- Looking for include files CMAKE_HAVE_PTHREAD_H
-- Looking for include files CMAKE_HAVE_PTHREAD_H - found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Looking for XOpenDisplay in /usr/lib/libX11.so;/usr/lib/libXext.so
-- Looking for XOpenDisplay in /usr/lib/libX11.so;/usr/lib/libXext.so - found
-- Looking for gethostbyname
-- Looking for gethostbyname - found
-- Looking for connect
-- Looking for connect - found
-- Looking for remove
-- Looking for remove - found
-- Looking for shmat
-- Looking for shmat - found
-- Looking for IceConnectionNumber in ICE
-- Looking for IceConnectionNumber in ICE - not found
-- Found X11: /usr/lib/libX11.so
-- Configuring done
-- Generating done
-- Build files have been written to:
/home/andre/bsps/git_openkinect/libfreenect/build

11 - make

LTIB> make
Scanning dependencies of target freenect Scanning dependencies of target freenectstatic [ 10%] [ 20%] Building C object src/CMakeFiles/freenect.dir/core.c.o
[ 30%] Building C object src/CMakeFiles/freenect.dir/tilt.c.o
[ 40%] Building C object src/CMakeFiles/freenect.dir/cameras.c.o
[ 50%] [ 60%] Building C object src/CMakeFiles/freenectstatic.dir/core.c.o
Building C object src/CMakeFiles/freenectstatic.dir/tilt.c.o
Building C object src/CMakeFiles/freenect.dir/usb_libusb10.c.o
[ 80%] [ 80%] Building C object
src/CMakeFiles/freenect.dir/registration.c.o
Building C object src/CMakeFiles/freenectstatic.dir/cameras.c.o
[ 90%] /home/andre/bsps/git_openkinect/libfreenect/src/cameras.c: In function 'freenect_fetch_zero_plane_info':
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:914: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:915: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:916: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:917: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:918: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:919: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:920: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:921: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c: At top level:
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:749: warning:
'read_register' defined but not used
[100%] Building C object src/CMakeFiles/freenectstatic.dir/registration.c.o
Building C object src/CMakeFiles/freenectstatic.dir/usb_libusb10.c.o
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c: In function
'freenect_fetch_zero_plane_info':
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:914: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:915: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:916: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:917: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:918: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:919: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:920: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:921: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c: At top level:
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:749: warning:
'read_register' defined but not used
Linking C shared library ../lib/libfreenect.so [100%] Built target freenect Linking C static library ../lib/libfreenect.a [100%] Built target freenectstatic
LTIB>

12 - copy the lib and header files to your rootfs:

cp ../build/lib/* /ltib/rootfs/usr/lib
cp ../include/* /ltib/rootfs/usr/include

with this procedure, the openkinect is correctly installed on your rootfs. If you have any problems, for example, with the libusb, you should update the libusb package on ltib, it is easily done by copying the new downloaded package to /opt/freescale/package and changing the correspondent spec file at: ../ltib/dist/lfs-5.1/package_name/package_name.spec. Every package can be updated using this approach, after that you can rebuild the package using ltib commands:

./ltib -p package_name.spec -m prep
./ltib -p package_name.spec -m scbuild
./ltib -p package_name.spec -m scdeploy

Now as simple application, I used the last video to texture project, but now displaying the images from kinect device instead:

this demo application was wrote based on the sample applications that comes with the openkinect library. In the future posts I´ll give more details about the code.

Source Code: git@github.com:andreluizeng/Kinect-hack.git

EOF !