To count the pedestrians in the scenarios with the sparse or dense crowd, a network based on the improved Inception-ResNet-A module is proposed, which is trained with the gradient boosting method of ensemble learning, and the details of the proposed method are given. Besides, a dataset collected in a real scenario, which contains illumination and camera view changes, and other three public datasets are used to evaluate the robustness of the proposed method in terms of illumination, population density, and camera view changes. The experimental results show that the proposed method is robust to the aforementioned changes. In addition, the proposed method favorably outperforms the state-of-the-art approaches in terms of accuracy and stability.