What is the cost to file a case against a university in andhra pradesh high court

You cannot file the proposed case  at the Andhra Pradesh High Court in Amaravathi. Even though you live in AP and received the emails there, Indian courts strictly look at where the "cause of action" arose and where the defending institution is located. Because IIM Kozhikode is a central institution physically located in Kerala, and their admission committee operates and makes decisions there, the High Court of Kerala (in Ernakulam) holds the proper territorial jurisdiction for a Writ Petition under Article 226 of the Constitution.

The court fee and the costs would be let known by the chosen advcoate.  You can engage an advocate practicing at Kerala high court. You do not have to go to Kerala, you can have the dealing and terms of the advocate.

Many advocates handle filings digitally through the e-filing portal and argue matters via virtual video conferencing. Look for a reliable lawyer who practices at the Kerala High Court. A good lawyer will likely seek an interim stay or an urgent direction from the court asking IIM Kozhikode to keep one seat vacant or provisionally accept her marks while the petition is being heard. Please consult with a legal professional practicing in the Kerala High Court as soon as possible to prevent the admission window from closing entirely. 

Contact the Principal of IIM Kozhikode personally tto consider the case of your daughter sympathetically before seeking any legal remedy, which can be availed in Kerala (local jurisdictional courts) only.

AP High court shall not entertain any case/ writ on the subject matter as explained in detail by learned expert Mr. T Kalaiselvan, which I concur and appreciate his acument.