We present DetectFusion, an RGB-D SLAM system that runs in real time and can ro-bustly handle semantically known and unknown objects that can move dynamically inthe scene. Our system detects, segments and assigns semantic class labels to knownobjects in the scene, while tracking and reconstructing them even when they move in-dependently in front of the monocular camera. In contrast to related work, we achievereal-time computational performance on semantic instance segmentation with a novelmethod combining 2D object detection and 3D geometric segmentation. In addition, wepropose a method for detecting and segmenting the motion of semantically unknown ob-jects, thus further improving the accuracy of camera tracking and map reconstruction. We show that our method performs on par or better than previous work in terms of lo-calization and object reconstruction accuracy, while achieving about 20 fps even if theobjects are segmented in each frame