At 19 Duncan Bannatyne was behind bars. Today the entrepreneur and Dragons' Den star has amassed £136m with shrewd investments in ice creams vans, care homes and gyms.

His next venture, to stave off the tedium, is in housing.

Duncan Bannatyne doesn't like the new manager at his private members club. He especially doesn't like him today after being told that the photographer won't be allowed to take pictures to accompany this interview. Who says? Sherri says. "Sherri," barks Bannatyne in his clipped Glaswegian accent. "Who the hell is Sherri?" The unfortunate go-between in this game of Chinese whispers, a quivering bar manager, eventually plucks up the courage to explain that Sherri is a switchboard operator … and the Bannatyne wrath is unleashed. Hell hath no fury like a millionaire scorned.

After a tirade that begins, "I've been a member here for five years, you go back to your manager and tell him …", a tray of Earl Grey arrives and this seems to calm everyone down. Bannatyne, though, has been biting his tongue, and over the course of the interview, calculates his next move.

Portrait by Eva Vermandel

Bannatyne is a SMEM, a self-coined acronym he uses to describe himself and his rich list buddies. It stands for self-made entrepreneur millionaire. The 56-year-old has reached SMEM status after spending the past 20 years building up a Bannatyne empire that includes care homes, fitness clubs, nurseries, bars, hotels and casinos. In the past couple of years, he has also become something of a housewives' favourite after appearing in the BBC reality show, Dragons' Den. The Den, as it is known to the fans, sees hopeful businesspeople pitching moneymaking ideas to four successful entrepreneurs - the eponymous dragons. Straight-talking Bannatyne quickly became the star and the most feared Dragon thanks to his acerbic putdowns. "Doing Dragons' Den is great. I am never cruel. Just honest and fair," he says.

Bannatyne likes to keep busy and he sure as hell likes to make money, which is why he is moving into residential development. Nibbling on a pistachio shortbread biscuit in the decadent tearoom, he explains why his latest venture, Bannatyne Housing, is going to be a success, how he gets away with doing five hours' work a week, and why building up his £136m fortune was the easiest thing he has done in his life. "I put my mind to it and eliminated the obstacles standing in my way. Anyone can become a millionaire. It's so easy, it's boring," he says.

The making of a millionaire

Bannatyne grew up on a council estate in Clydebank, Glasgow, one of seven children. The privations of his working-class upbringing made him bitter. He says he hated anyone who had more than him and became obsessed with wealth. When he was 19, he joined the Royal Navy, where his obsession soon landed him in hot water. For a £5 bet, he tried to throw his commanding officer over the side of an aircraft carrier. He was dishonorably discharged and sentenced to nine months in a military jail.

He spent most of his 20s "sowing wild oats" until, at 29, he decided to become a millionaire. Lying on a beach in Jersey, he turned to his then girlfriend and said he was fed up with being poor and that he was going to move back to England and make a million.

He returned to the mainland and, in 1980, bought an ice-cream van for less than £500. Within six years, he had generated enough business to fund a fleet of vans that he sold for £25,000. With the proceeds he started up Quality Care Homes, a successful business with 36 developments across the country. In 1992, he floated the company on the stock exchange, which valued it at £18m. Bannatyne took home £13m - he'd made it.

He then shocked friends and colleagues by ploughing everything into a new venture, a chain of fitness clubs. Tony Bell, Bannatyne's project director for 10 years, remembers how astonished he was when his boss revealed his risky plan.

"I'd been working with Duncan for about nine months when he sold Quality Care Homes," he says. "I couldn't believe it when he told me he was going to put all of it into a new venture. I thought, ‘This guy is totally mad'. But look where we are nine years on. Now he has £136m."

Bell adds that hardly anyone of sound mind would have taken the same risk, which is why Bannatyne is a multimillionaire and the rest of us are not. "You can count the number of people who would have done the same as Duncan on one hand," he says.

The fitness clubs were a massive success. Bannatyne had tapped into a market that nobody thought would take off in the UK. Today, there is a health club or gym on every corner, health food stores on most high streets and fitness DVDs lining the shelves of newsagents. He has spent the past 15 years investing in the right businesses and it seems unlikely that he will slip up now. With that in mind, he explains how he is going to take his own advice to crack the housing industry.

Bannatyne Housing

Bannatyne is planning developments on four sites. Three will be built on spare land next to Bannatyne fitness clubs. These sites are in Dumfries, Mansfield and Chafford Hundred, a commuter development in Essex. The fourth is in Stockton-on-Tees. "I'd been considering developing houses for a couple of years.

With all the spare land around the health clubs, I could see that if I did it right, developing it would make a bigger profit than selling," he says.

The current dip in the housing market is not putting him off. Indeed, he argues that it's the ideal time to invest: "When the market is booming, it's impossible to find contractors, joiners, plumbers - getting a workforce together is hard. During a slow time, it's much easier. As soon as we got permission to build, we started things moving. Why delay? Because of a dip in the market? There's no reason to."

So supremely confident in his abilities as a businessman, Bannatyne has started up his housing company with no expert advice or guidance. "I've been involved in the construction industry since 1986 when the care home business took off, I know what I'm doing. I'm not going to do anything different. I've never done that in my whole career and I've made a lot of money. so why would I change my approach now?"

When he was 19, he joined the Royal Navy, where for a £5 bet he tried to throw an officer over the side of an aircraft carrier

The company will stick to conventional designs, according to Andy Roberts, Bannatyne's architect and design director at Newcastle-based practice Waring & Netts, although he emphasises the developments will be well upmarket. "It's all about quality." Most of the properties, particularly the flats in Chafford Hundred, are going to be Duncan Bannatyne all over: chic, sleek and suave.

"There will be additional features in these apartments, particularly in the kitchens and bathrooms. Luxurious finishes, high-quality white goods, ceramic floors, fitted carpets and broadband connections. Basically, no cutting corners," says Roberts. Each development has been designed to be individual, from the detached houses in Dumfries to the modern apartments in Essex.

Despite the fact that the first Bannatyne home will not be available for viewing until May, he is certain that he will sell every property. He has studied the market and says he knows what people want to buy. "In Dumfries, there is a housing estate beside the one we are developing that is very similar to what we will be doing in that area and I know that all those places went pretty fast." The Essex apartments are all designed for one or two people as Bannatyne is buying into the "independent living" trend.

"I know people are living alone because I live alone. There's no doubt it's happening. In Essex in particular, there is just nobody moving into that part of the country with families so there is no point in building a load of family houses there."

If any Bannatyne property looks like it may not sell, he has trick up his sleeve. "People who buy a Bannatyne home will have the added bonus of getting a free membership to the on-site Bannatyne fitness club," he laughs.

Working for the man

He is not just sure of his product, he is sure of his staff. He has surrounded himself with a team of experts whom he trusts unequivocally. Project director Tony Bell says his boss certainly knows how to delegate. "Duncan loves to set up businesses but I am not sure he likes running them. That's why he hired me, because I get my buzz out of building buildings and changing the landscape. One of Duncan's qualities is that he isn't precious, he can trust other people with his hard work."

He's quite the perfectionist, though, and expects his team to have the same high standards. For his first six years with Bannatyne, Bell worked 12-hour days and only had one weekend off a month. "I often said to Duncan, ‘I'm working three people's jobs,' and he would just look at me and grin. He knew who he'd hired and why. He knew that if there was work to be done, he could rely on me to do it."

Of course, Bannatyne still keeps a sharp eye on the progress of every one of his ventures and the housing company will be no different: "I will be down on those sites checking the workmanship. It will have to be of a very, very high standard. If you are paid to do a job like plumbing or joining, then you will damn well have to do it right."

The predicted turnover of the company remains a mystery. Bannatyne says it will all depend on how quickly the properties sell, but it could be quite a moneyspinner: he paid nothing for three of the sites and the units in Chafford Hundred and Dumfries will go for about £200,000 and the Mansfield homes will sell for £60,000.

Bannatyne remains calm about his latest venture's potential impact. "I am so small at the moment. I have no huge plans. We'll just see how things go," he says.

He certainly doesn't think he's in a position to shake up the industry. "I think the housing industry will see me as a tiny speck of dust on the back of an elephant." But architect Roberts thinks Bannatyne Housing is going to be bigger than his boss is letting on. "It's just a few sites but over time I think Bannatyne properties will evolve into a niche product and will be incredibly popular. All you have to do is look at Duncan's track record."

Portrait by Eva Vermandel

Being Duncan

Bannatyne seems relieved as the interview steers away from the housing company towards his social life. He sits upright on the purple chaise longue, and his words become harder to catch as he talks faster and with more gusto. Duncan Bannatyne may be a businessman first and foremost but the motivation behind what he does is what brings him the most happiness. "I have a passion for money, but I don't just make it, I spend it," he laughs.

Bannatyne's qualities as a businessman are the same traits that give him the freedom to enjoy his fortune. He follows his own advice, sticks to what he knows works and he delegates - freeing up his own time. He shatters any illusions that he is a nose-to-the-grindstone type. "My businesses take up hardly any time at all. I work about five hours a week," he says with a hint of pride. "I spend the rest of the time meeting people, lunching, going to dinners and parties. It's a fantastic life. I love it."

On the way out, the manager is loitering in the entrance hall. Bannatyne can't resist. He strides over to the man and barks: "Why can we not take the pictures in here? I have been a member here for five years." The manager clasps his hands behind his back and puffs out his chest. "Because the communications company said no and now I am saying no and that is final," he spits back. Bannatyne is quiet for a moment, seemingly defeated, until: "I don't believe you. I don't believe the communications team would say ‘no' to free advertising."

With that, Bannatyne turns on his heel and marches through the heavy oak door, collapsing into schoolboy giggles as soon as he is outside. "Sorry about that," he gasps between peals of laughter. "That was brilliant."

In the millionaire’s chair

The worst business idea I ever heard was … On Dragons’ Den, someone came up with cardboard beach furniture. For goodness sake. What if it rained? Where do you sit when you have just come out of the sea and are soaking?

My favourite song is … I’m tone deaf so I don’t like music – sad isn’t it? I have one song that brings back a memory though. It’s Tie a Yellow Ribbon. When I came out of military prison, my girlfriend said she would meet me under the clock at Euston station. When I got there, the station was packed and there was more than one clock. I was pushing through people, searching for her and eventually found the clock that I was sure she meant. I was stood there waiting and waiting and that song came into my head. With all that suspense, I felt like the bus driver waiting to turn the corner to see if there was a yellow ribbon tied around the old oak tree. She eventually appeared and we went off and had a great weekend together!

The worst thing you can do to make a bad impression in business is … Turn up late or smoke before you come to a meeting. Nine times out of 10, I can tell if someone has been smoking before they come to see me or are in my presence and it is a big no-no. I hate it. I think it’s the stupidest thing in the world.

If I lost everything tomorrow I would … Start again of course.

The saying ‘money can’t buy you happiness’ is … Stupid. You can’t go down to Marks & Spencer and buy half a bucket of happiness and money doesn’t automatically make you happy, but it gives you options and freedom and lets you lead a good life.