This paper addresses the problem of performance analysis based on the communication modeling of large-scale heterogeneous distributed systems, with an emphasis on enterprise Grid computing systems. The study of communication layers is important, as the overall performance of a distributed system often critically hinges on the effectiveness of this part. We propose an analytical model that is based on probabilistic analysis and queuing networks. The proposed model considers the processor as well as network heterogeneity of the enterprise Grid system. The model is validated through comprehensive simulations, which demonstrate that the proposed model exhibits a good degree of accuracy for various system sizes, and under different working conditions.