First of all, because he doesn't want to be an official anymore. Because of his great ability, the emperor was suspicious of him, because the ancient emperors were very suspicious. As long as their servants have some influence, they are afraid that they will take away their throne. If he continued to be an official at that time, he might be worried about his life, so I hope he can leave soon.

Secondly, because another person has grown up and can take his place, he has now become a useless person to the emperor. In fact, in ancient times, the emperor's courtiers were easily replaced, because as long as another person could take his place, the former person would definitely be eliminated, and with the growth of that person's age, the emperor would pay less and less attention to him. At that time, he just felt that he was too old to be so useful to the emperor, and now there is someone who can take his place. Under such circumstances, he will definitely be eliminated by the emperor.

Finally, I was afraid of seeing Yongzheng. In fact, he had made up his mind to leave at that time, but he was afraid that Yongzheng would not let him go and upset his plan. So he told others that he needed to leave quickly, otherwise it would be too late, because if Yongzheng came, he would definitely not be allowed to quit the officialdom, but if he did not quit the officialdom, his life would be threatened. Even if Yongzheng didn't want to let him go at that time, he felt sure not to go.

In short, the reason why he said it was too late at that time was that he was afraid that Yongzheng would not be able to leave when he came.