Kevin Myers, the HSE's chief inspector of construction, said that the new figures offered "no reassurance" for the industry.
He said: "They do not provide evidence of the cultural change in attitudes and standards across the industry in line with the commitments given at the 2001 Health and Safety Summit."
He added: "Much still needs to be done to improve the standards of risk awareness and management across the entire industry."
Four members of the public were killed in 2003/04, compared with five the previous year. The number of workers killed on site fell from 71 to 68.
Falls from height still cause by far the greatest number of accidents, rising from 33 in 2002/03 to 38 in the past year.
The provisional construction fatality figures will be verified by officials and published in July, before which the HSE is planning another "blitz" on building sites.
The HSE plans to target falls from height, workplace transport and safe lifting.