Self-Supervised 3D Scene Understanding

I will present our recent work on how a general AI algorithm can be used for 3D scene understanding to reduce the need for training data. More exactly, we propose several modifications of the Monte Carlo Tree Search (MCTS) algorithm to retrieve objects and room layouts from noisy RGB-D scans. While MCTS was developed as a game-playing algorithm, we show it can also be used for complex perception problems. Our adapted MCTS algorithm has few easy-to-tune hyperparameters and can optimise general losses. We use it to optimise the posterior probability of objects and room layout hypotheses given the RGB-D data. This results in a render-and-compare method that explores the solution space efficiently. I will then show that the same algorithm can be applied to other scene understanding problems with RGB data only.