Deep Camera Pose Regression Using Pseudo-LiDAR

An accurate and robust large-scale localization system is an integral component for active areas of research such as autonomous vehicles and augmented reality. To this end, many learning algorithms have been proposed that predict 6DOF camera pose from RGB or RGB-D images. However, previous methods that incorporate depth typically treat the data the same way as RGB images, often adding depth maps as additional channels to RGB images and passing them through convolutional neural networks (CNNs). In this paper, we show that converting depth maps into pseudo-LiDAR signals, previously shown to be useful for 3D object detection, is a better representation for camera localization tasks by projecting point clouds that can accurately determine 6DOF camera pose. This is demonstrated by first comparing localization accuracies of a network operating exclusively on pseudo-LiDAR representations, with networks operating exclusively on depth maps. We then propose FusionLoc, a novel architecture that uses pseudo-LiDAR to regress a 6DOF camera pose. FusionLoc is a dual stream neural network, which aims to remedy common issues with typical 2D CNNs operating on RGB-D images. The results from this architecture are compared against various other state-of-the-art deep pose regression implementations using the 7 Scenes dataset. The findings are that FusionLoc performs better than a number of other camera localization methods, with a notable improvement being, on average, 0.33m and 4.35° more accurate than RGB-D PoseNet. By proving the validity of using pseudo-LiDAR signals over depth maps for localization, there are new considerations when implementing large-scale localization systems.

While FusionLoc is competitive with other camera pose regression methods, we believe there is still room for improvement and further testing. We can incorporate muti-scale grouping in the SA layers for more robust features. Furthermore, it may also be beneficial to add temporal constraints to learn features consistent throughout pointsets and images, similar to MapNet. Finally, we believe FusionLoc can produce even better results with improved depth maps. 7 Scenes was collected using the Kinect v1, however, since then, depth cameras and depth estimation techniques, including monocular depth estimation, have seen significant improvements. We would like to further test FusionLoc using depth maps generated using varying techniques and tools.