We present a computationally efficient RGB-D based pose estimation solution for less computationally resourced MAVs, which are ideally suited as members in a swarm. Our approach applies the sufficient statistics derived for a least-squares problem to our problem context. RANSAC-based outlier detection in aligning corresponding feature points is a time consuming operation in visual pose estimation. The additive nature of the used sufficient statistics significantly reduces the computation time of the RANSAC procedure since the pose estimation in each test loop can be computed by reusing previously computed sufficient statistics. This eliminates the need for recomputing estimates from scratch each time. A simpler hypotheses testing method gave similar performance in terms of speed but less accurate than our proposed method. We further increase the efficiency by reducing the problem size to four dimensions using attitude data from an Attitude and Heading Reference System (AHRS). Using a real-world dataset, we show that our algorithm saves up to 94% of computation time for the RANSAC-based procedure in pose estimation while improving the accuracy.