Budget? Fudge it!

Good engineering is not always good economics. The Sydney Opera House and the Channel Tunnel are great feats of engineering, but the financial track record of both is lousy. The opera house was completed 10 years late and 14 times overbudget, and traffic in the opening year of the Anglo-French tunnel was one-fifth of planners' forecasts. More recently, British taxpayers footed the bill when the Scottish parliament's price tag rose from £40m to £338m – and that cost is still rising. In fact, the world is full of buildings, bridges and tunnels that delight engineers, but would appal any financier who cared to look at the numbers.

In 1980, Sir Peter Hall, now professor of planning at University College London, highlighted several examples in his book Great Planning Disasters. Describing the decision-making process behind schemes including the Sydney Opera House, Concorde and San Francisco's light rail network, he detected a pattern: contractors and bureaucrats conspiring to spend public money on grandiose schemes that the public did not need. He looked at political behaviour through the lens of market economics and saw politicians competing to sell their products in return for votes. In turn, voters are consumers looking for the optimum package of public goods and services.

The villains of the piece are the bureaucrats, who aim to maximise the size of their bureaucratic empires by encouraging wasteful public spending. They deliberately underestimate costs and overestimate utility to get their projects approved, then team up with construction firms who rely on big public sector contracts. For example, several engineering firms backed the campaign for San Francisco's rail network, and were paid to build it once it got the go-ahead. The bureaucrats' task is made easier if they work for a politician suffering from "monument complex" – John Joseph Cahill, the prime minister of New South Wales who commissioned the Sydney Opera House, wanted to bequeath a grandiose landmark to the city.

Hall spins a good yarn, but his sample of projects is too small to make generalisations. However, a recent study of 258 transport infrastructure projects led by Professor Bent Flyvbjerg at Aalborg University in Denmark confirms many of his suspicions, and the findings are statistically robust. The projects in Megaprojects and Risk: An Anatomy of Ambition cover Europe, the USA, Japan and developing countries, dating between 1927 and 1998.

They show a consistent pattern: on average, the final cost is 40% higher than the estimate at the time of decision to build. Very few come in under budget.

Since cost estimates appear not to have improved over time, the authors conclude that public officials are unwilling to learn from their mistakes. In fact, they contend that they are not mistakes at all, but deliberately unrealistic estimates. The World Bank, which funds many infrastructure projects in developing countries, describes the practice of basing cost estimates on a best-case scenario as "appraisal optimism". Flyvbjerg calls it lying.

Moreover, the authors show that traffic forecasts are highly unreliable. Actual passenger traffic on the rail projects they studied averaged 39% less than forecast. For many projects, such as the Channel Tunnel, passenger traffic is down more than 80%. Just as Hall claimed 20 years earlier, the authors conclude that politicians and bureaucrats deceive themselves into thinking that projects are economically justified. If cost and use estimates were more realistic, many projects would never make it beyond the drawing board.

The villains are bureaucrats who encourage wasteful public spending

The study recommends three main remedies: greater transparency, better risk management, and private finance for public infrastructure projects. The first two are common sense, although hard to achieve where conservative bureaucracies are involved. But the most radical suggestion, involving increased use of PFIs and PPPs, is more open to argument.

The authors maintain that if at least one-third of a project's cost is covered by private finance unprotected by government guarantee, then only economically viable projects will go ahead. In theory, bankers will only back projects that can generate a steady revenue stream, and they will apply pressure to ensure projects stay on budget.

The budget-busting Channel Tunnel was backed by a consortium of financiers and contractors whose main motivation was winning contracts. The solution, say the authors, is to keep financiers and contractors separate.

But there are other problems with private finance. Some big public projects have tremendous spin-offs – for example, the Sydney Opera House and San Francisco's Golden Gate Bridge attract tourists who boost the local economy. However, private financiers are not moved by such considerations: since the projects derive little revenue from the tourists, they will not take them into account in deciding whether or not to fund the project. What's more, governments can borrow money more cheaply than private companies, so using private finance makes projects more expensive to fund.

Many commentators suspect the real motive behind PFIs is to take capital costs and risk off the government's balance sheet, and some mutter that PFI is just government-sanctioned "Enron accounting". At any rate, it seems logical that the transparency sought by the study's authors should extend to transparency about why governments are using private finance at all.

Finally, the private sector is not immune to self-deluding optimism. The tendency to underestimate costs and overestimate revenues is particularly acute when new technology or grand geopolitical gestures are involved. If anyone doubts that this is the case, consider the hare-brained investment decisions taken during the dotcom boom.