Point-Based Fusion for Multimodal 3D Detection in Autonomous Driving

Abstract

In the broader field of mechanical technology, and specifically within the domain of self-driving vehicles, cameras and LIDAR are crucial sensor modalities that provide complementary information, offering significant potential for sensor fusion. However, directly merging multi-sensor data through point projection can lead to information loss due to quantization, and managing the differences between data formats from multiple sensors remains a challenge. To address these issues, we propose an new fusion method that leverages continuous convolution, point-pooling, and a learned MLP to achieve superior detection performance. Our approach integrates the segmentation mask with raw LIDAR points instead of using projected points, thereby avoiding quantization loss. We conduct neighbor searches on the points and retrieve corresponding semantic features from images to concatenate image and LIDAR data. Subsequently, we apply continuous convolution, point-pooling, and a learned MLP to obtain the fused output. The pooling and aggregation operations, as extensions of convolution, are specifically designed to handle the disparities in data formats. Our detection network is divided into two stages: in the first stage, preliminary proposals and segmentation features are generated; in the second stage, the fusion result with the segmentation mask is refined to produce the final prediction. Our method aims to achieve precise object detection in 3D environments by enhancing LIDAR point data with semantic features from images, allowing for the flexibility to alternate segmentation sub-algorithms as needed. Extensive experiments on the KITTI dataset demonstrate the effectiveness of our approach, which achieves high precision and robust performance in 3D object detection tasks.