Oxford council's planning committee rejected designs for a hotel, restaurant and underground car park at the £14m Oxford Prison development last week.
It said the two-storey design would overshadow one of the few remaining parts of Oxford Castle, a mound dating back to the 11th century.
The building is the second controversial Jeremy Dixon.Edward Jones project in Oxford in recent years.
The first was the £45m Said Business School for Oxford University.
John Goddard, chair of the planning committee, said all its members were opposed to the scheme, and that the building should be totally redesigned.
Goddard said: "The committee was not particularly struck by the design; the scheme went to public consultation and it received a very negative response."
He added that the building was too close to the roadside.
Councillors want a heritage centre based on St George's Tower, another remaining feature of the castle, to be given equal prominence on the site.
Sir Jeremy Dixon, a partner in Jeremy Dixon.Edward Jones, said of the criticisms: "If people don't like it, they don't like it. There's not a lot I can do about that.
"We had an idea and a plan, but people are entitled to their opinions, and our arguments are not totally watertight."
It is understood that Dixon and the developer, Osborne Properties, are to meet planning officers next week.
The Said Business School was rejected in 1997 because of opposition to development on one of the last green spaces in the centre of Oxford. The plan was finally approved in 1999 and the scheme is due to be completed next month.
Goddard said the complaints about the castle building have coloured the public's perception of the business school.
He said: "Currently, because of the negative comments surrounding the prison site development, the public has started to become very rude about the design of the business school."
