Wayne Erasmus said his son Huw, 31, moved without notice from a unit in Carmarthen to Birmingham and then on to St Andrew’s Healthcare in Northampton.
The Care Quality Commission (CQC) has raised concerns and criticised repeated failings in St Andrew’s leadership.
St Andrew’s said it had new leaders in place committed to making improvements.
Mr Erasmus, of Hendy, Carmarthenshire, said both St Andrew’s and his local health board, Hywel Dda, had told him Huw did not want to speak to him.
“That’s what they say but I know Huw gets worried and when he’s worried he says he doesn’t want to talk to the family,” Mr Erasmus said.
“But then when he does want to speak with his family he has to wait two weeks for his fortnightly review… but they put pressure on him then to stick with no contact with the family. In my opinion, they’re hiding behind that.”
Huw, who was raised in Hendy, Carmarthenshire, has been held under the Mental Health Act for six years. He was originally supposed to be there for six months.
“When he was in Carmarthen I could hear his voice when they asked him if he wanted to speak to mam or dad. Now that he’s in St Andrew’s – total black out,” Mr Erasmus said.
“I don’t think the place is suitable for him and the more I hear, the bigger the horror story is getting.”
Jane Haines’ daughter Ayla, who has anorexia, lives at the site. Ms Haines, from Carmarthen, said she was not allowed to contact her daughter as often as she would like and was worried about the level of care.
“The staff really do a phenomenal job and they’re fighting a losing battle. It’s down to the management… the ward at the very least should be sufficiently staffed and a lot of the time it isn’t.
“We’re restricted to four phone calls a week but for the last 18 months it’s been three 10-minute phone calls a week – it’s hell. That’s what kept her going was our contact, that was her reason for living, so it’s like, you’ve taken that away from her as well.
“There’s restrictions on what we can talk about. We’re not allowed to speak about medication, we’re not allowed to speak about the staff. I’m not allowed to give her hope and say ‘we will get you back to Wales’.”
CQC inspectors, who visited the charity’s headquarters in October, found the use of physical restraint had increased despite a plan to reduce it.
They also found the process of telling a patient’s family when something went wrong was “not fully effective”. Inspectors also found staff were not always confident to raise concerns without fear of reprisals.
St Andrew’s said it was “working closely with NHS Wales” to ensure patients received the best and most appropriate care.
“There are often multiple factors that need to be taken into account, which at times require that these vulnerable individuals need very specialist treatment and observation in a safe, secure setting,” it said.
Hywel Dda health board said it took concerns about patients’ care and treatment “very seriously” and reviewed any concerns or feedback with its partners.