If you do not want to stay with him and do not want to continue marriage relationship, you can file divorce case on the available grounds. Regarding custody of the children, as they are with you, you need not file any case. If he wants to get the custody, he will file the case. You can contest the case. The court will decide the question - with whom the children should live - taking into consideration of the welfare of the children. So, do not worry about the custody of the children. The other parent, who does not get custody will get visitation rights.