The news comes a month after the publication of the urban white paper, which pledged to relaunch the troubled initiative.
The resignation has also sparked a review of the advisory panel by government regeneration agency English Partnerships, which administers the villages.
The news of Lord Rogers' resignation comes as Prescott is due to open the first four homes at
the £250m Millennium Village in Greenwich, south-east London, today. The flagship scheme is 18 months behind schedule.
The second Millennium Village in Allerton Bywater, West Yorkshire, is also facing renewed difficulties, because of concerns over the suitability of the site.
A spokesperson for EP confirmed Lord Rogers' departure. The spokesperson said: "In the light of Lord Rogers standing down, the role of the advisory panel is under review." EP said Lord Rogers had resigned because he wanted to cut back on his commitments after his appointment as an adviser to the Greater London Authority.
However, it is understood that Lord Rogers was frustrated with the slow progress of the first two villages. A member of the advisory panel said: "There haven't been any meetings this year, he hasn't had much input, so he thought the thing would not collapse without him."
In the light of Lord Rogers standing down, the role of the advisory panel is under review
English Partnerships spokesperson
The panel also includes CABE chairman Sir Stuart Lipton, urban taskforce member Ricky Burdett, architect David Mackay and Mark Whitby, founder of Whitby Bird & Partners.
A mentoring group set up last September to police Miller/Gleeson, joint developers of Allerton Bywater, on the design, chaired by Burdett, has not met for nine months.
The scheme has weathered a storm of local opposition and doubts by the developers over its financial viability. The opposition intensified after floods struck the lower reaches of the village.
The scheme has now been submitted for outline planning permission, although local consultation is continuing.
A source close to the project said: "There were strong political reasons for its selection, but more homework needed to be done to make sure it all stacked up financially. It was a classic problem of top-down versus bottom-up decision making."
Another advisory panel member speculated that Lord Rogers might have resigned to allow Richard Rogers Partnership to compete for one of the next competitions without a conflict of interest.