My grandmother signed a release deed releasing her rights of her inheritance to her brothers for a consideration. now the cheques tgat were mentioned in the deed got dishonoured. can she file for cancellation of release deed

She has both the options, viz; can proceed to cancel the RD and / or file a case under the provisions of section 138 / 142 of the Negotiable Instruments Act, 1881 

Priority is for filing a case for dishonoring of cheque, by issuing legal notice and initiating proceedings within time frame.  Immediately contact a local advocate and issue such notice to the brothers of your grandmother.  This verdict helps in the cancellation of the sale deed at a later stage.