With the development of artificial intelligence technology, the research of galaxy morphology classification using deep learning methods has made great progress, but there are still shortcomings in classification accuracy, automation and spatial characteristics representation of galaxies. The Vision Transformer model has good robustness in galaxy morphology classification, but has limitations in handling multi-scale images. In this paper, we propose to introduce the Feature Pyramid Networks (FPN) into the Vision Transformer (ViT) model to classify galaxies. The results show that the average accuracy, precision, recall, and F1-score of the FPN-ViT model are above 95%, and the indexes are improved compared with the traditional ViT model. Meanwhile, we add different levels of Gaussian noise and pretzel noise to the original galaxy images to verify that the FPN-ViT model can obtain better classification performance for low signal-to-noise ratio data. In addition, to evaluate the model comprehensively, the t-distributed Stochastic Neighbor Embedding (t-SNE) algorithm is used to visualize and analyze the classification results, which can show the effect of FPN-ViT model on galaxy morphology classification more directly. The application of FPN network to the classification of galaxy morphology by ViT model is a new attempt, which is of great importance for the subsequent research.