He says that London's transport, tourist and sporting facilities are not equipped to handle the games – even if a national stadium is built.
Kaufman says London is already too congested and the government's commitment to help upgrade the city's infrastructure will not be enough because the costs will be too steep for chancellor Gordon Brown.
"Horror novels have been written about London seizing up completely one day. That day could arrive with the Olympics," he says.
"Unless hundreds of millions, if not billions, of pounds of public money are put into such infrastructure, then the real problems of access will not be solved and even a successful Olympic bid could turn out to be a fiasco."
Earlier this year consultant Arup was appointed by the government, the Greater London Authority and the British Olympic Authority to provide cost analysis of a London bid.