Dowry , that is a different issue which is not raised by the bride. Now the issue raised by op is the groom's family made the bride's family spend around 1 lakh for the engagement which is only after acceptance of the bride by the groom is ensured and the girl's family has heeded to every demand made by the boy. Inspite of all this, stepping back is unacceptable and the girl's family is entitled for damages for sure.
Op you have not stated whether the cash of 2 lac is returned or not and if there is any video footage evidencing the engagement and giving of cash.
Its very weird to hear that there are rumours about the boy from the boy's family themselves,when such an issue has to be raised by the girl's family if there is any such rumour.
You have a good case so go ahead engaging a good lawyer.