No, it is not haram for you to stay away form them, but deal with them diplomatically. Still Stay polite with them or you will lose your husband to side with them even if you are truthful, if you react to their acts. The most thing that kills them is when you don’t react and when you don’t argue with them! This frustrates them a lot. They would become happy only when you react, shout, argue with them or become angry. Ignore what they say or accuse you with a big smile and pretend that you don’t hear what they Say!
Seek help form Allah and stay dutiful to Allah and your husband. May Allah guide them and us all.