Image synthesis is a critical task in various computer vision technologies, and lots of methods tried to translate semantic images into realistic ones for controllable synthesis. With the increasing image resolution, networks are becoming larger, and applications of related methods are restricted. To alleviate the problem, we propose a lightweight mutable network for semantic image synthesis. The network is based on generative adversarial networks. We introduce the feature pyramid architecture to the generator and reduce the hidden node numbers. We also design a mutable scheme where the pyramid will involve fewer layers for smaller images. To improve the performance of the lightweight generator, we further propose a weighted discriminator and a refined loss. The experiments on several public datasets show that our method is effective and achieves competitive performance, proving that a small network can also achieve high-quality semantic image synthesis.