Liam Grafton made 47 calls to the woman on the telephone and intercom, at the residential complex on Chatsworth Street, between 10am and 12noon, on January 25.
She called colleagues and put him on speakerphone when he said he wanted to have sex with her, said prosecutor Donna Fawcett.
In a statement, the manager said: "I did feel quite scared. I was on my own and I knew he was in the building."
Later on, Grafton knocked on the office door, but when he was ignored, he shouted: "Why are you hiding? I'm not a threat."
He hit the UPVC door, causing a five or six inch crack to appear, then put a note under it which said: "Shut the door, you can't hide."
The court heard a similar incident had happened a couple of years previously. He was last in court in 2010 for a different matter.
Chris Perry, mitigating, said Grafton had been a resident at the complex for people with mental health and alcohol problems for six years, "with no problems."
He said there had been a problem with the intercom system buzzing in Grafton's room every time a resident used it, and he became frustrated.
"He was exceptionally intoxicated on the day," Mr Perry said. "He is exceptionally embarrassed."
He said that Grafton had been drinking heavily since the death of his baby daughter, and there is "a heavy reliance on cider, lager and occasionally vodka."
He failed to turn up to court on April 23.
Probation officer Cheryl Nisbet said he had been drinking all night and had been trying to sleep the next morning.
"Because he was annoyed he decided to annoy staff," she said.
Grafton, 37, of Chatsworth Street, admitted harassment without violence, criminal damage, and failure to surrender, when he appeared at Mansfield Magistrates Court, on Wednesday.
A one year restraining order, not to contact the support worker except at work, was made.
A 12 month community order was imposed with a six month alcohol treatment requirement, as well as six rehabilitation days to explore his attitudes towards women.
He must pay £300 towards the cost of the door, but no court costs were ordered because of his means.