It is narrated from Hazrat Abu Zubair Makki that I saw the Ka'bah become empty of those performing tawaf; after the Fajr prayer and after the Asr prayer, no one would perform tawaf.

Imam Malik said that whoever started tawaf, then the call to prayer was given for Fajr or Asr prayer, he should perform the prayer with the Imam, and after the prayer, he should complete the tawaf, but he should not perform the two rak'ahs until the sun rises or sets, and if he performs them after the Maghrib prayer, then there is no objection.

Imam Malik, may Allah have mercy on him, said: If a person performs one tawaf after the Fajr or Asr prayer and delays the two rak'ahs until the sun rises, as Sayyiduna Umar, may Allah be pleased with him, did, or until the sun sets, then there is no harm in it. Now, he should perform the two rak'ahs of tawaf as soon as the sun sets or after the Maghrib prayer.