Growth stood at 8% last year but a weakening private sector market will bring it down to 1.9% this year. Output is predicted to fall to a low of 1% next year before rising slightly to 1.1% in 2005.
CPA chief executive Michael Ankers said that 2002 had been the industry's best year since 1988, but he noted that the industry was reliant on government spending to avoid stagnation.
He said: "Our forecasts show that the construction industry will be very much dependent upon the government's investment plans for its growth over the next three years, with public sector growth, including PFI, forecast to increase by 6.3% this year, slowing to 5.7% in 2004 and 4.3% in 2005."
Ankers said the government had failed to hit its spending targets in the past two years and that this had led to doubts over its ability to deliver the promised capital spending programme. Government investment undershot its target by £2.6bn last year.
Private commercial output is forecast to slow sharply over the next three years because of falling demand for office space and overcapacity. Few developers are willing to build speculative commercial schemes at the moment.
The cooling of the housing market and slower house price inflation is also likely to reduce the number of starts on new housing.
Starts are expected to fall from 167,000 units this year to 155,000 in 2005.
In the infrastructure market, output rose 9% in 2002, largely driven by work on big projects such as the Birmingham toll road and the Channel Tunnel Rail Link.
Road activity is set to moderate this year, as there is a lack of projects to fill the gap left by the completion of the toll road.
The industrial sector is set to pick up between 2002 and 2005. Last year, the sector's output fell by 10.7%, but should see a return to growth by 2005.
This should be partially driven by a recovery in the construction of warehousing and distribution facilities.