Time-of-Flight sensors and stereo vision systems are two of the most diffused depth acquisition devices for commercial and industrial applications. They share complementary strengths and weaknesses. For this reason, the combination of data acquired from these devices can improve the final depth estimation performance. We introduces a dataset acquired with a multi-camera system composed by a Kinect v2 ToF sensor, an Intel RealSense R200 active stereo sensor and a ZED passive stereo camera system. The acquired scenes include indoor settings with different external lighting conditions. The depth ground truth has been acquired for each scene of the dataset using a line laser. The data can be used for developing fusion and denoising algorithms for depth estimation and test with different lighting conditions. A subset of the data has already been used for the experimental evaluation of the stereo-ToF fusion method of Agresti et al.

Data Collection

The multi camera acquisition system used to acquire the proposed dataset is arranged as in the figure below. The reference system is the ZED camera in the center, underneath the ZED there is the Kinect and above there is the RealSense R200. The three cameras are kept in place by a plastic mount specifically designed to fit them. The depth camera of the Kinect is approximately horizontally aligned with the left camera of the ZED with 40 mm vertical displacement, while the color camera is approximately in between the passive stereo pair. The RealSense R200 is placed approximately 20 mm above the ZED camera, with the two IR and color camera inside the baseline of the passive stereo pair.

The subjects of the 10 scenes in the REAL3EXT dataset try to stress various flows of the stereo and ToF systems. Critical points are for example lack of texture for the passive stereo system and the presence of low reflect elements and external illumination for the active sensors. The scenes are composed by flat surfaces with and without textures, plants and objects of various material such as plastic, paper and cotton fabric. These are characterized by various specularity properties as reflective and glossy surfaces and rough materials. Each scene was recorded under 4 different external lighting conditions, which are the following: with no external light; with regular lighting; with stronger light; with an additional incandescent light source. Each lighting condition can highlight the weakness and strength of the different depth estimation algorithms. We added the acquisitions with the additional incandescent light source since its spectrum, in the IR wavelength, covers the working range of the active depth cameras and it is a known problem for those devices.