If they are accusing you with adultery and you have people to testify in court, you can go to the Islamic court and file a slander case against them. This is your right.
But if they are only talking and gossiping without accusing you of anything, there isn’t much you can do except to warn them that you will take from their good deeds on the Day of Judgment.
Don’t occupy your mind by thinking in revenge. There are not even worth it.
It is said that if people kick you, this is a sign that you are in the front and people only throw stones at the fruitful trees.
Be confident and ignore what they say as Allah will make you take from their good deeds on the Day of Judgment what satisfies you