David Cameron and Boris Johnson recently shared a helicopter ride together. As the current and wannabe leaders of the Conservative party clattered through the sky, the British prime minister leaned over to the man who aches to replace him and remarked: “If this helicopter crashes, I suppose it will be George and Michael and Theresa in the run-off.”

That he now feels so relaxed that he can crack jokes about leadership contests, and make them to one of the rivals who has most nakedly lusted after his crown, is one illustration of a change in the atmosphere of the Tory party.

Just a few weeks ago, Westminster bubbled with intrigue about backbench plots to collect the signatures necessary to launch a challenge to his premiership. The home secretary was one of those who bared the teeth of their leadership ambitions, which they would not have done had they not smelled prime ministerial blood in the water.

The nadir was around the time of the Queen’s speech when Cameron could not even persuade a significant chunk of his party to vote for the government’s programme.

I was always sceptical that a challenge would actually materialise, but the talk of one troubled Number 10 because it was not entirely fantastic. So febrile was the state of their party, senior Tories grew anxious that they might be on the brink of implosion. Leading Lib Dems wondered whether they needed to start making contingency plans for the combustion of their coalition partners precipitating an early general election.

All has now been transformed. Superficially at least. Talk of challenges has evaporated in the summer heat. The prime minister can head for his holidays feeling secure. Conservative MPs seem more united and optimistic than they have been for some time.

Various reasons explain this mood change. The bulk of his party has rallied around the prime minister’s position on a European referendum, at least for now. The furies are subsiding over some of the other issues gay marriage being the most notable example that have pulled them apart. Memories of the Tory losses in the local elections and their drubbing at the hands of the Lib Dems and Ukip at the Eastleigh byelection have somewhat faded.

Some of his colleagues give a lot of credit to Cameron for boosting morale. Long accused by his backbenches of being chilly and aloof, he has been making a determined effort to woo his troops.

A recent hog roast for Tory MPs in the back garden of Number 10 was just one of many attempts to make them feel loved — or at least not entirely ignored — by their leader. The reshuffle that was to have been held last week was postponed until the autumn rather than risk spoiling the newly harmonious atmosphere.

In the final few prime minister’s questions before the recess, Cameron has turned in some more than usually belligerent performances. “Every day this country is getting stronger and he is getting weaker,” he shouted at Ed Miliband during their parting encounter, indicating the Tory intention to make the next election campaign personal and presidential.

I doubt this sort of formulaic abuse sounds terribly edifying to the outside world, but it appeals to the primeval instincts of Tory MPs who get a rush from seeing their man clubbing the chief of the rival tribe.

Crime is down to the lowest levels in three decades. Immigration numbers are moving in the direction that Tories desire. Abu Qatada has been deported. The exposure of some appalling failures by the health service play to their advantage — or so Tories calculate. Few of them think that the NHS, usually the worst issue for the Conservatives, can be turned into an election-winner.

What they are aiming to do with their attacks on Labour’s stewardship of the NHS is to even up the score between the parties and neutralise health as an election battleground. In this assault, they have been joined by their allies in the media. The Tory press, which has been bitterly estranged from Cameron over the Leveson inquiry and the threat of regulation, has been moving back on side.

Conservative MPs have also become more chipper about their prospects because they think the economy is finally looking up. Indicators of business confidence have grown more positive. Unemployment is down, though overall levels remain horribly high.

The IMF has nudged up its estimate of Britain’s growth prospects. These are the reasons for this surge in Conservative confidence. Yet has anything fundamental changed? Or is it more an illustration that the party now has the personality of a manic depressive? Some senior Tories worry it might be the latter.

At a Number 10 garden party last week, one member of the cabinet observed: “Some of my colleagues have swung from irrational despair to irrational exuberance.” A party that can lurch so rapidly and dramatically from profound gloom to heady exhilaration is not psychologically stable. One worry for Cameron is that it wouldn’t take much for the mood to swing back just as violently.

The economic recovery is fragile, which is why George Osborne is being careful to downplay expectations. The resumption of growth is also very patchy. One member of the cabinet remarks: “In so much as there is a recovery, it is mainly confined to the southeast of England.”

Tory passions about Europe are currently sedated, but experience suggests that this may well not last. Many senior Conservatives still expect Ukip to do very well in next year’s Euro elections and quite possibly come out top. Some Tories have got excited by a few opinion polls suggesting that they are closing the gap with Labour and one reporting that they’d pulled up to level pegging.

Yet these movements can more plausibly be explained by margins of error and differences in sampling methods than by a substantial shift in the public mood. Labour’s lead is soft, but you have to say it has been pretty consistent.

It has oscillated around nine to 10 points for some time. The last published poll that put the Tories ahead, and only by a slight margin, was in March 2012.

The phone-hacking trials begin in September. They threaten to be a daily drip-feed of embarrassment for Cameron as the world is reminded of his intimate links with former Murdoch editors: Rebekah Brooks, his horse-riding companion, and Andy Coulson, whom he judged fit to bring into Downing Street. Another large question mark about the prime minister’s judgement has been raised by his employment of Lynton Crosby without apparently having asked either himself or Crosby whether the lobbyist’s long and controversial client list was going to be a problem. Cameron sounds slippery when he says that he was never “lobbied” on behalf of big tobacco by his campaign chief while not actually denying that they have had conversations about cigarette packaging.

The case for Crosby advanced by senior Tories is that almost any price is worth paying because he is such a sharp strategist. “Brilliant”, in the estimation of the mayor of London with whom he worked on two election campaigns. What does this brilliance boil down to? Telling the Tories to hone aggressive messages tightly focused on traditional themes such as welfare and be intensely negative about Labour. It seems to me that Cameron is paying a big fee for some rather obvious advice. Not every senior Tory is convinced that the ‘Wizard of Oz’ is any more magical than the original. While accepting that he has helped to raise their short-term tactical game, some quietly question whether it is a long-term strategy for winning an election.

The Conservative party’s structural weakness has not changed. That problem is finding a way to appeal to enough voters to secure a parliamentary majority, something at which the Tory party has failed in every election since 1992. On some calculations, Labour can win a majority in the Commons with just 35 per cent of the vote. It would be a lousy way to win, but it would be a win. The Conservatives need a lead over Labour of around seven points and at least 40 per cent of the vote to have any confidence of winning a majority next time. That is three points more than the vote share they achieved at the last election and no Conservative prime minister has improved his party’s popularity since 1955.

It is not just history that is stacked against them. So is demography. As extensive research by Michael Ashcroft has illustrated, the Conservatives have very little support among ethnic minority voters who are now a key constituency in many of the suburban marginals. Then there is geography. To win a majority, the Tories would have to do better in northern England — they have given up on Scotland — than they did in 2010 against a long-serving and unpopular Labour government.

Two years out from the next election, it remains extremely hard to see how the Conservatives will win an overall majority. Which is why the more sober of their number believe that their best chance of remaining in power probably lies in another hung parliament and another coalition.

— Guardian News & Media Ltd