Under existing rules, unlisted dwellings can be altered without consent.
Raynsford said the government was considering tightening the law even further to outlaw permitted development rights.
Under this rule, homeowners in conservation areas are allowed to make changes to unlisted dwellings without planning consent.