Sensors (Basel). 2019 Feb 19;19(4). doi: 10.3390/s19040866.

Exploring RGB+Depth Fusion for Real-Time Object Detection.

Sensors (Basel, Switzerland)

Tanguy Ophoff, Kristof Van Beeck, Toon Goedemé

PMID: 30791476 PMCID: PMC6412390 DOI: 10.3390/s19040866

Abstract

In this paper, we investigate whether fusing depth information on top of normal RGB data for camera-based object detection can help to increase the performance of current state-of-the-art single-shot detection networks. Indeed, depth sensing is easily acquired using depth cameras such as a Kinect or stereo setups. We investigate the optimal manner to perform this sensor fusion with a special focus on lightweight single-pass convolutional neural network (CNN) architectures, enabling real-time processing on limited hardware. For this, we implement a network architecture allowing us to parameterize at which network layer both information sources are fused together. We performed exhaustive experiments to determine the optimal fusion point in the network, from which we can conclude that fusing towards the mid to late layers provides the best results. Our best fusion models significantly outperform the baseline RGB network in both accuracy and localization of the detections.

Keywords: Depth; Neural Networks; Object detection; RGB; RGBD; Sensor fusion; Single-shot

References

1. IEEE Trans Pattern Anal Mach Intell. 2008 Feb;30(2):328-41 - PubMed
2. IEEE Trans Pattern Anal Mach Intell. 2014 Aug;36(8):1532-45 - PubMed
3. Resour Conserv Recycl. 2018 Aug;135:323-334 - PubMed

Publication Types

• Journal Article