Because when the person praying turns his attention here and there during his prayer, Allah Almighty also turns His blessed face away from the face of the one praying.

Sayyiduna Abu Dharr (may Allah be pleased with him) narrates that the Messenger of Allah (peace and blessings be upon him) said likewise.

Sayyiduna Abu Dharr (may Allah be pleased with him) narrates that the Messenger of Allah (peace and blessings be upon him) said: "Allah Almighty continuously pays attention to His servant as long as he does not turn away, but when the servant turns his face away (removes his attention from the prayer), then Allah Almighty also turns His attention away from him."

Sayyiduna Harith Ash'ari (may Allah be pleased with him) narrates that the Prophet Muhammad (peace and blessings be upon him) said: "Indeed, Allah the Exalted commanded Sayyiduna Yahya bin Zakariya (peace be upon him) to act upon five things and to command the Children of Israel to act upon them as well. Therefore, he used to admonish the people regarding these matters. Then he said: 'Verily, Allah the Exalted has commanded you to perform prayer. So when you turn your faces (towards prayer), then do not turn here and there, because when a servant prays to Allah the Exalted, Allah turns His Face towards the face of His servant, and He does not turn His Face away from him until the servant turns his face away.'"