What a laugh we have all had at the Russians' expense. They have apparently been caught red-handed spying on America, with secret agents, passwords, codes, woodland drops and invisible ink. At a London breakfast on Tuesday addressed by Sir John Sawers, head of the British Secret Intelligence Service, the assembled top brass were full of beans. They do things differently here.

They had been to trooping the colour, an annual military pageant where Britain's foes are deterred with up-to-the-minute stallions, breastplates, drums and waving swords. They'll have Johnny Taliban quaking in his flip-flops in next to no time.

The saga of the Russian spy ring is yet more evidence that whatever defence spending is about, it has nothing to do with defence. The FBI and the CIA have bust an operation that must have cost the Russians millions and yielded nothing that could not have been gleaned from the New York Times, Washington Post and political blogs. Why not leave the spies at it? I am sure they were paying tax. It is laughable that they posed any threat to the American people.

Clearly old habits die hard. There must be George Smileys in Russia's federal security service who date from the great days of the KGB. They mutter that the ancient methods are the best. Not for them unreliable newfangled electronics. Just put a boot on the pavement. Never trust the internet when you can use a beer-bottle drop. You never know when you will need an agent in place. It is like the macho detective in the satire Last Tango in Aberystwyth: "Louie needed a druid — and needed one fast". The spy ring was merely Russia's trooping the colour, a touch of the old days to keep the lads on their toes.

You never can tell

Spies love to assert that, while 90 per cent of secret service activity is barking mad, you can never tell which is the other 10 per cent. James Bond might seem no more than a sadistic clotheshorse, but once in a while he saves the planet. Hence the motto of security services down the ages: "You never can tell". Or as that modern philosopher Donald Rumsfeld would put it: "The unknown unknowns are what get you in the end."

While most defence spending these days is bonkers, I can see the case for secret intelligence, especially the independent variety in which the British and American security services claim to deal. The likelihood of Russia or North Korea or Pakistan or even Iran launching a nuclear attack on Britain is so infinitesimal as to be trivial. But a modest outlay to keep the government up to speed on aggressive lunatics in those countries is an insurance premium worth paying.

For 20 years since the Cold War the security services, like the armed forces, have been seeking a new role when their chief reason for existing has gone. The world is a vastly safer place.

Soldiers think they can guard children against drug traffickers, ships against pirates, dirty bombs against proliferation and states against failing. They cannot. All they can do is try to make people more afraid. Americans are susceptible to being scared witless by the defence lobby. Britons are commendably resistant.

The most effective protection I have from a terrorist bomb in my street is a competent intelligence service and a police force not dancing to the home secretary's (interior minister) tune but plugged in to neighbourhood street life. The coalition can spend all it wants on defence kit, but it will not make me feel safe, not like a police force that knows each nook and cranny of a locality and can tell when something is nasty in the woodshed.

Any competent government needs good intelligence, in the widest sense of the word. Spies are part of that. They are not burdened with the historical baggage of the armed forces and their consequent, often fanatical, conservatism. They can redeploy with ease, formulate their own objectives and speak truth to power in private and without political constraint.

But effectiveness depends on two factors — secrecy and independence. Secrecy has been partly blown by the eagerness of both the domestic British Security Service, known as MI5, and overseas SIS to be publicly accountable and "avowed". I could never see the point. Secret means advice given and received secretly, otherwise "intelligence" is just another competing input to policy, its content inevitably leaking into the public domain.

The corruption of intelligence by politics, by rulers who hear only what they want to hear, is such a familiar hazard of espionage as no longer to be an excuse for failure. But the risk of corruption must rise when intelligence is doctored against the possibility of parliamentary and press scrutiny, and when its independence is compromised by closeness to ministers (and their spin doctors).

Spies like to protest that they fail in public, but must succeed in private. That is the fate of all who work in defence and security. But having compromised their secrecy, they have a credibility problem in pleading for support.

Simon Jenkins is a journalist and author. He writes for The Guardian and The Sunday Times, and broadcasts for the BBC.