The committee criticised the scheme for its lack of "basic Urban Design and architectural concepts" and called for more work on the quality of spaces proposed and the built form.
The RHWL team is proposing to build a 93,000 m2 tower at Rumford Place, the site of CABE's top-floor office in Elizabeth House.
Under plans prepared by RHWL, this 16-storey tower, and a lower office block that extends down York Road past the Shell Building, will be demolished.
The plan is to cover over what is now York Road with office space and turn the low-rise building into a road past the Eurostar terminal at Waterloo. A 33-floor tower with a sail-like design has been mooted as the landmark building.
Peter Stewart, CABE's design review head, said the reviewers were happy with the development concept but were unsure how the scheme would fit into the area.
He said: "This is an enormous project and much more needs to be done on how it fits into the immediate area and the city as a whole. At the moment we are underwhelmed."
The report said the plans conveyed little idea of how the building would look in its surroundings. It noted that the documentation for other tall buildings, such as Swiss Re in the City, had included plans that gave a "largely accurate" impression of how the building would look.
It said: "The planning application material is rather skimpy by comparison with other recent projects of this size and prominence. The detailed design work has the flavour of being a technical or problem-solving exercise rather than an attempt to make real an idea about what the building should be like."
The report noted: "Our own view is that buildings of such prominence on the skyline demand excellence both in urban design and architectural terms, and that this scheme has not yet demonstrated that it succeeds on either ground."
Bill Edgerley, P&O's developments director, said the CABE's approval was vital to getting the scheme though planning. He added there was always a debate over how much detail to include in the planning application.