Habré, who ran the country from 1982 to 1990, was convicted in 2016 by a court in Senegal of crimes against humanity and war crimes. He was sentenced to life imprisonment.
Clement Abaifouta, who is one of the victims, told BBC's Newsday that it would be unfair for Habré to be pardoned.
"I think that Hissène Habré's wife is simply trying to create some kind of compassion to try and make people accept a potential pardoning that is not what we the victims have ever wanted. We've never wanted for Hissène Habré, who was sentenced to jail for everything he's done during his time in office, to benefit from any kind of pardon," he said.
Habré's wife Fatime Raymonne told Radio France Internationale last month that the former president's health is deteriorating and accused prison officials of ignoring him when he fell in a bathroom.
According to Mrs Habré, her husband notified her using a phone that was given to him after he suffered a heart attack and it was after the call that prison authorities heeded to Habré's calls.
Reed Brody, a lawyer from Human Rights Watch who worked with victims in the case, told Newsday he is sceptical of Mrs Habré's plea.
"When he fled Chad he literary emptied out the treasury and he used that money to buy influence and protection in Senegal and that's in fact why it took 25 years to get him to court in the first place. Now those same supporters are pressing on, they are saying he's 77-years-old, he fell in the shower, Mrs Habre has been on the front pages of Senegalese newspapers asking for his release on compassionate grounds," he said.
Habré was convicted of rape, sexual slavery and ordering killings and ordered to pay up to $33,000 (£27,000) to each of his victims or their surviving relatives.
According to Mr Brody, Habre has hidden his money under the names of relatives making it hard to trace his finances and the victims have never been paid the judgement money.
- Countries: Africa