Imam Malik, may Allah have mercy on him, was asked: If there is fragrance on a garment, and then its scent disappears, is it permissible to wear it in the state of Ihram? He said: Yes, as long as the color of saffron or wars does not remain on it.
