A GA-based pruning fully connected network for tuned connections in deep networks

Deep neural networks have proven themselves as a strong approach in image classification and object detection with high accuracy. However, they are computationally demanding and the trained networks contain millions of active parameters and connections. Two recent trends of having deeper and dense architectures and the deployment of trained networks on resource-constrained devices such as smart phones and portable tablets bring new challenges. Instead of deploying an ensemble of smaller networks, we propose a pruning methodology on a trained network so that a smaller version of a fully trained network has the same and even better accuracy in comparison to the original one. We achieve two objectives with the pruning scheme. First, we have a smaller network with a better accuracy level, and we make the trained model avoids overfitting. Accordingly, an evolutionary based framework including three steps is defined to perform further tuning on trained deep network using dropping nodes and connections. This study shows that implementing genetic algorithm, after preprocessing and training stages, not only results in partially connected networks, but also increases performance and reduces overfitting specially when the depth and width of fully connected networks are investigated in small datasets.