README.md

Kinect Depth Image Smoothing

• "Kinect Smoothing" helps you to smooth and filter the Kinect depth image and trajectory data.

How to Use it

• Installation

git clone https://github.com/intelligent-control-lab/Kinect_Smoothing.git
pip install -r requirements.txt

• Running

  Please check example.ipynb and kinect_preprocess_example.py.

• Data preparation
  We saved many frames of depth images in data/sample_img.pkl and saved corresponding frames of position coordinates in data/sample_pose.pkl
e.g. sample_img = [ [ image_1 ] , [ image_2 ], ... , [ image_t ] ]. each image_i has a shape of (width, height).
  In case if anyone wants to use it for multiple files, then the code below should work.

import joblib 
import cv2
import glob 

X_data = []
file_lists = glob.glob("/*.bmp")  # image path
for files in sorted(file_lists):
    if files.endswith(".bmp"):
        image = cv2.imread(files)
        X_data.append(image)
        joblib.dump(X_data, 'image_frames.pkl')

Features

1 . Depth Image Smoothing

• Hole Filling Filter

  In the original Kinect depth image, there are many invalid pixels (they appear black on the image and are registered as 0's). This function helps you to fill the invalid values with based on the valid pixels in the vicinity. The methods for hole filling are as follows:

  • "min": Fill in with the neighboring minimum valid value
  • "max": Fill in with the neighboring maximum valid value, refer to A computationally efficient denoising and hole-filling method for depth image enhancement
  • "mean": Fill in with the average of all neighboring valid values
  • "mode": Fill in with the mode of neighboring valid values, refer to Smoothing Kinect Depth Frames in Real-Time
  • "fmi": Fast Matching Inpainting, refer to An Image Inpainting Technique Based on the Fast Marching Method
  • "ns": Fluid Dynamics Inpainting, refer to Navier-Stokes, Fluid Dynamics, and Image and Video Inpainting

• Denoising Filter

  After pixel filling, Denoising filters can be used to improve the resolution of the depth image. The methods for denoising filling are as follows:

  • "modeling": filter with Kinect V2 noise model, refer to Modeling Kinect Sensor Noise for Improved 3D Reconstruction and Tracking
  • "modeling_pf": another Kinect V2 noise modeling by Peter Fankhauser Kinect v2 for Mobile Robot Navigation: Evaluation and Modeling
  • "anisotropic": smoothing with anisotropic filtering Scale-space and edge detection using anisotropic diffusion
  • "gaussian": smoothing with Gaussian filtering

2 . Trajectory Smoothing

• Crop Filter:

  The x, y coordinates of the trajectory were captured by some object detection algorithms (e.g. Openpose). Sometimes the object will be positioned on the background, and the depth coordinates might register as invalid values on the Kinect. The Crop-Filter crops the invalid value and run some interpolation methods to replace it. The methods for invalid value replacement are as follows:

  • conventional interpolation methods: such as "zero","linear","slinear","quadratic","cubic","previous","next","nearest".
  • "pchip": PCHIP 1-d monotonic cubic interpolation, refer to Monotone Piecewise Cubic Interpolation
  • "akima": Akima 1D Interpolator, refer to A new method of interpolation and smooth curve fitting based on local procedures

• Gradient Crop Filter:

  Similar to Crop-Filter, the GradientCrop_Filter crops the large gradient values between near pixels maybe miss-labeled as background. The methods for invalid value replacement are as follows:

  • conventional interpolation methods: such as "zero","linear","slinear","quadratic","cubic","previous","next","nearest".
  • "pchip": PCHIP 1-d monotonic cubic interpolation.
  • "akima": Akima 1D Interpolator.

• Smooth Filter:

  Smooth the data with a specific filter, which is effectively reducing the anomaly in the trajectory series. The methods for smoothing filter are as follows:

  • "kalman": smooth the signal with Kalman filter, refer to Fundamentals of Kalman Filtering
  • "wiener": smooth the signal with Wiener filter
  • "median": smooth the signal with median filter
  • "moving_average": smooth the signal with moving average filter
  • "exponential_moving_average": smooth the signal with exponential moving average filter

• Motion Sampler:

  Motion detection and filtering out the stationary part of the trajectory. (Significant movements are preserved.)

3 . Real World Coordinate Calculation

• Coordinate Calculator

  Transforming the pixel level coordinate of Kinect image to the real world coordinate.