You are not allowed to attack them if they swear at you. This is what they want and this is what the media needs to fuel people against Islam.
If you were attacked, you are allowed to defend yourself without exaggerating. So if someone slaps you on the face, you are not allowed to poke his eyes. You may slap him on the face as well.
Defend yourself but always have the upper hand and don’t give them the means to attack Islam.