According to the original novel, Cao Cao was not entirely scared off by Zhang Fei. The other reason was that Cao Cao was suspicious. When he saw the dust in the woods behind Zhang Fei, he thought there was an ambush and did not dare to act rashly. Zhang Fei shouted three times and frightened Xia Houjie to death, so Cao Cao retreated in panic. As for what Zhang Fei said,

The first sentence: "I am Zhang Yide from Yan! Who dares to fight to the death with me?"

The second sentence: "Zhang Yide from Yan is here ! Who dares to fight to the death? "

The third sentence: "Fight but not fight, retreat but not retreat, but why!"

For full details, you can read the original version. In Chapter 42 of the novel, Zhang Yide makes a big fuss at Changban Bridge.