Graves announced he was stepping down as Yellows chief with immediate effect just days after their play-off failure at Hinckley.
But he admits the timing of his departure was not how he intended his time at the club to end.
“It is not how I wanted to leave the club,” said Graves. “I would have liked to go to the new ground, but I know it’s being built and that is good enough for me.
“I would like to have stayed, but with some of the events over the weekend, it made me question whether it is worth it.
“It is my decision. I will make it myself, the club didn't make it for me. The club were shocked, the players were shocked.
“The players’ reaction made me want to think again, but if I am going to do something I have to do it 100 per cent or it's not worth it.
"I am retiring after 15 years in football working alongside Stevo, (assistant Paul Stevenson), and also working with Phil Henry (first team coach), having been involved in running youth teams, reserves, AFC Hucknall and the first team over those years.
“I am walking away completely and will not be involved in any capacity other than to maybe watch some of the lads who stay on.”
But Graves believes he is leaving Town in much better shape than when he first came to the helm.
“There were 30-40 fans when we took over and we thank them for that support,” he added. “Without them we don't have a club. If there's no past there is no now.
“We have done more than well. The plan was to start again, get the club over to the new ground and next year was going to be my last season.
“We went into the game on Saturday confident, but it didn't go our way.
“We didn’t get the rub of the green, but that's not why we lost.
“We didn't defend when we had to but I'm proud of the lads. We had a good season and progressed.
“We set out originally to get to the play-offs and to see where we are, afterall this was our first season at this step.
“I have achieved everything I set out to do. Going back to those people who put all the money in to pay off the tax debt, then we came in during the AFC days and the debt has been run up again.
“My ambition, along with Phil and Paul, was to keep the club stable and keep the club going so that all the people who pledged the money all those years ago was not wasted.
“When the ground is built we will know all the effort was not wasted.”
But Graves feels that the move away from Watnall Road has come at a cost with Town not having the on-field financial power of some of their rivals.
“I was getting texts after the (Hinckley) game from players who want to come next season,” he revealed.
“It’s ok them wanting to come, but the issue is can you afford them?
“I’ve always said that while I was in charge, and knowing how the club is run, you can only progress naturally without throwing money at it and destabilising the club.
“We always knew the ground was here, but it has come at a cost and the club has not progressed as fast as some people would like.
“People have to know the whole story and they don’t and that’s the way it is. I have always been open for people to come and chat with me if they choose to.
“We fell short on Saturday, but you only have to look at the infrastructure of how they are set up and that is what you are competing against.”
He leaves Town with many fond memories with taking over at the club and winning promotion, two particular highlights for Graves.
He added: “I have good memories of the Matlock game when we won the league and also taking over the club.
“I took over when ex-pro footballers had turned the job down and the club was in a pickle.
“I have met a lot of good people. There are too many people to thank.
"It’s been an incredible time and I’d like to thank the players over the seasons for their support, the chairman and management committee, all of our fantastic supporters and Liz Morley for giving me the chance to be manager back in those dark days that we hope never to see again.
But most of all a big thank you to Stevo and Phil for everything they have helped achieve, their support and above all their loyalty.
“I have had good captains. Stevo has been with me 15 years, we used to fall out like I don't know what in youth football.
“We got together and look what happened. We have done it together and there are not any people who last as long as we have at one club.
“We didn’t have a big influx of players and we took it steady.
“I made lots of mistakes, as we all do, and I’m the first to admit it. I made mistakes on Saturday, but that is football.
"The messages I have received from players (past and present), opposition managers and even other club supporters have been overwhelming. I wish nothing but further success for the club, the new Manager and our supporters in the new era they are about to embark on.
Town's Chairman Bob Scotney praised the work down by Graves over the last decade.
"Andy started in 2013 with the First Team and with no budget and the club in financial trouble,” he said. “He has done a great job for the club with improved results season upon season.”