Some millisecond pulsars rotate very stably, which can provide an independent time benchmark called pulsar time based on remote natural objects and last for millions or even billions of years. The pulsar time is characterized by the high stability, long operation time and wide service range. In order to reduce the influence of Gaussian noise on pulsar time construction in millisecond pulsar timing observation, a construction algorithm of ensemble pulsar time based on bispectral filtering is proposed. Using the observation data of four millisecond pulsars released by IPTA (International Pulsar Timing Array) to establish a pulsar time, the stability of ensemble pulsar time in different time scales is analyzed. Then, it is compared with the four atomic clocks which make up the international atomic time. The results show that the bispectral filtering algorithm can effectively reduce the influence of observation noise and improve the stability of ensemble pulsar time. The long-term observation data of four millisecond pulsars (PSR J0437-4715, J0613-0200, J1713+0747 and J1909-3744) are processed. Compared with the classical weighted algorithm, the one-year and ten-year stability of ensemble pulsar time are raised to 1.50times10^-14, 3.50times10^-16 from 7.77times10^-14, 8.56times10^-16 respectively, and the stability of single pulsar time also shows the same rule. It is also found that the five-year stability of the ensemble pulsar time is better than that of the atomic clock TAI (PTB2) of the German Physikalisch-Technische Bundesanstalt, so the pulsar can be used to improve the long-term stability of atomic time.