Self Supervised Learning for Multiple Object Tracking in 3D Point Clouds
- Aakash Kumar University of Central Florida
- Jyoti Kini University of Central Florida
- Ajmal Mian University of Western Australia
- Mubarak Shah Univerity of Central Florida
Abstract
Multiple object tracking in 3D point clouds has applications in mobile robots and autonomous driving. This is a challenging problem due to the sparse nature of the point clouds and the added difficulty of annotation in 3D for supervised learning. To overcome these challenges, we propose a neural network architecture that learns effective object features and their affinities in a self supervised fashion for multiple object tracking in 3D point clouds captured with LiDAR sensors. For self supervision, we use two approaches. First, we generate two augmented LiDAR frames from a single real frame by applying translation, rotation and cutout to the objects. Second, we synthesize a LiDAR frame using CAD models or primitive geometric shapes and then apply the above three augmentations to them. Hence, the ground truth object locations and associations are known in both frames for self supervision. This removes the need to annotate object associations in real data, and additionally the need for training data collection and annotation for object detection in synthetic data. To the best of our knowledge, this is the first self supervised multiple object tracking method for 3D data. Our model achieves state of the art results.
Video
Proposed Network Architecture
In self-supervised learning, detected objects in a single LiDAR frame (real or synthetic) are cropped to create frame \( A \) without background. Objects in \( A \) are augmented with translation, rotation, and cutout to generate frames \( B \) and \( \Gamma \). Fixing the number of points per object to \( p = 256 \), a \( 3 \times p \times N \) matrix (where \( N \) is the number of objects) is formed to represent a LiDAR frame which is then passed through the object feature extractor that consists of five convolution layers followed by max-pooling along the \( p \) dimension. A box-feature extractor comprising 3 convolution layers extracts box features which are concatenated with object features. Combined features of \( N \) objects from two LiDAR frames are permuted to form an \( N \times N \times (576 \times 2) \) matrix \( \mu \) which is passed through five convolution layers to gradually decrease the depth of \( \mu \) and compute an \( N \times N \) affinity matrix \( M \). All convolution kernels in SS3D-MOT network are of size \( 1 \times 1 \). To account for objects entering and leaving the scene, \( M \) is augmented with an extra column and row. Finally, row-wise and column-wise soft-max are performed to get \( M_C \) and \( M_R \) matrices used to compute the loss.
Synthetic Lidar Frames
Synthetic LiDAR frames generated from (a) CAD objects (ModelNet-40 data), and (b) primitive shapes (cones, cubes etc). We place the objects at different locations in 3D space to create a synthetic LiDAR frame which is then augmented to form two frames with known ground truth movements of the objects. For augmentation, we use translation, rotation and object cut out (to simulate occlusion).
Qualitative Results
Quantitative Results
Results from the JRDB Leaderboard. Our model achieves the best MOTA, IDF1 and HOTA scores on the leaderboard using self supervision with synthetic frames generated from CAD objects.
Results of our model on the validation sets of KITTI and JRDB datasets, when trained in supervised v/s self supervised setting.
Ablation study for self supervision: The effect of different augmentations when our model is trained on the real JRDB dataset or synthetic LiDAR frames.