After considerable attempts on monocular visual odometry, we have shifted to using a depth sensor which is expected to give accurate visual odometry compared to a monocular camera. So, the first step would be setting up the Kinect v2.
Kinect v2 has got a good support on Windows through the Kinect SDK but doesn’t have a good support on Linux. Rovers at Italian Mars Society and spacesuits at Austrian Space Forum are already running on Ubuntu 14.04 and it would be recommended to implement the project on a Linux machine. Linux does yet have a stable and standard support for Kinect v2 but there are a couple of options available. Below are the libraries that can be considered for hacking Kinect v2.
After examining all the libraries, libfreenect2 gave good results and it’s usability with Linux pushed me to continue with it. It can be generally used with C++ but Pylibfreenect2 provides a python interface for it.
Pylibfreenect2 is well documented and even has a good support on Github. There was an installation issue with the latest release because of pyx files and the issue was fixed within a day after I filed an issue on the Github. Installing from the source fixed the issue right away and they even tagged a new release within a day. Getting such a good support provides huge motivation to work. Installation instructions have been clearly documented here and executing a example script gives the below frames as output.
Calculating the registered frames has also been simple. Below are the registered color frame and the undistorted depth frame from Pylibfreenect2.
(Sorry for the bed being messy 😛 )
The black bars that are on the top and the bottom of the registered frame are expected and it’s because of limitations of the algorithm used. There is small issue regarding skipping some depth and RGB frames, with not Pylibfreenect2 alone but generally with all the drivers that are available for Kinect v2. Below is the log for the same.
It has not been tested but the issue exists with PyKinect as well. I couldn’t figure out a way to completely eleminate this but by using OpenGL or OpenCL, comparativetly less frames are skipped. There has also been a significant improvement by even reducing number of cv2.imshow statements. Once you decrease the number of frames that you want to display, the number of frames that would be skipped is comparatively reduced.
PyKinect has also been tested and below is the output of an example script which shows the body frame. It was fun seeing the stickman of myself.
There are also some apps available on windows store such as 3D Scan to play with. TBH, I couldn’t reconstruct any good 3D model but you should definetely try it out for fun.