This paper presents the first photometric registration pipeline forMixed Reality based on high quality illumination estimation byconvolutional neural network (CNN) methods. For easy adapta-tion and deployment of the system, we train the CNN using purelysynthetic images and apply them to real image data. To keep thepipeline accurate and efficient, we propose to fuse the light esti-mation results from multiple CNN instances, and we show an ap-proach for caching estimates over time. For optimal performance,we furthermore explore multiple strategies for the CNN training.Experimental results show that the proposed method yields highlyaccurate estimates for photo-realistic augmentation