Your grandmaa is alive or your father then send notice from their name via lawyer to the person who is harassing you, also clear that when it is not your house whether your father or grandmaa has objection of keeping the women in their house. and they are knowing this.
if he had issued notice on the name of your father then u no need to worry, also if you are sc/st then plz mentioned that in your notice as well,
send reply via notice that, without knowing the fact about the owner of house or having any kind of oral conversation asking about the fact details , he had send notice in your name as he is in hurry to encroch the place and harass both you and the women and both familiy members. if women is minority also menion that. just remember send reply to his notice within the timeperiod. keep material evidence ready toprove your reply ,
if they play loud music ,DJ parking vehicles any where near the path of the women and any sort of things mention that they are inventing new methods to harass the women and alternatively disturbing you and causing nuisance .he doesnot seem to talk and solve the issue so whithout checking thefacts he had send the notice to you. your layer would do the rest very good to shutdown his mouth. Good Luck!