Investors are turning their back on commodities as returns from futures, options and other derivatives fail to live up to expectations.

They heyday of commodities as a separate “asset class” appears over. In future, growth will be concentrated in physical arbitrage strategies, where trading houses and the trading arms of energy companies and utilities rather than banks and hedge funds have an advantage.

Banks tend to blame increased regulation and compliance imposed on them following the financial crisis for the reduced profitability of their commodity divisions. There is some truth in that. But the bigger problem is that growth in commodity trading has stalled as investors struggle to make money.

Following a large inflow of funds into the asset class during the boom of 2004-08, and then again during the Great Reflation of 2009 and 2010, investors have pulled back. Pacific Investment Management Company’s Commodity Real Return Strategy Fund, the largest investment vehicle in commodity derivatives, has seen its assets under management shrink 25 per cent to $14.3 billion (Dh52.5 billion) from $19 billion at the end of March 2011.

Schroders Alternative Solutions Commodity Fund has seen its assets halve from almost $5 billion in August 2011 to $2.6 billion at the end of February 2014. California Public Employees Retirement System, one of the earliest evangelists for allocating investments to commodities as a separate asset class, has cut its exposure to just $2.4 billion from more than $3.5 billion in early 2012.

Pimco, Schroders and CalPERS are some of the best-managed commodity funds in the business, yet all three recorded losses in 2013, underscoring just how difficult the environment has become. The downturn extends across the sector. Net long positions controlled by index-type investors in commodity futures contracts have fallen from $220 billion in January 2011 to $177 billion in January 2014, according to the US Commodity Futures Trading Commission.

Investors’ diminished appetite has cut deeply into the fees and other trading income available for the major investment banks and dealers. The average commodity fund lost more than 8 per cent of its value in 2013, compounding losses of 3 per cent in 2012 and 7 per cent in 2011, according to an analysis of Lipper fund data.

Major market benchmarks like the Dow Jones-UBS Commodity Index and the Standard and Poor’s Goldman Sachs Commodity Index have been flat or falling for three years. In a market where the spot price of most commodities has been stable, investors have not been able to earn enough return from taking on risk to cover the cost of storing and financing commodity stocks.

‘Facts and fantasies about commodity futures’, the 2004 research paper that popularised commodities as an asset class for a wider group of investors, promised an equity-like combination of risks and returns, as well as protection from inflation and diversification of a portfolio consisting of stocks and bonds.

But returns have proved disappointing. Commodity indices and hedge funds have lost money even as equity markets hit new post-crisis highs. The only diversification has been negative.

Protection against price rises has also come to seem less important as aggressive unconventional monetary policies by the major central banks have failed to spark inflation.

Waning enthusiasm for commodities therefore comes as no surprise, but it has been very painful for banks and hedge funds that rely on dealing and making markets in commodity derivatives, as their pool of potential business shrinks.

Interest has now shifted away from financial strategies based on futures, options and index swaps to physical market strategies that try to exploit small price differences related to the delivery time, location and quality of raw materials.

Physical arbitrage is the core business of trading firms like Vitol, Glencore, Trafigura, Mercuria, Cargill and Noble, which have correspondingly been able to expand their operations at the expense of the banks. But it is much harder for general financial investors to capture returns from physical arbitrage since it demands much more specialist knowledge and close relationships with both producers and consumers.

Even if some specialist funds promise to use their deep industry expertise to generate arbitrage-related performance, investors often struggle to prevent the fund managers and traders from appropriating all the returns. Banks too lack the economies of scale needed to exploit the ultra-thin margins in the arbitrage business.

The shift from financial to physical trading has therefore left banks, hedge funds and mutual funds struggling to generate acceptable returns and haemorrhaging their best traders and dealmakers to the trading houses and energy companies.

Market forces are shifting talent and capital to institutions and strategies that can employ them more effectively. Bank executives like to blame regulators for driving the business away with intrusive and expensive new rules.

But the underlying dynamics of the business have changed in ways which have pushed the banks and hedge funds to the sidelines and seem likely to keep them there in the short term.