Hedge funds are being forced to cut fees as investors pull out money amid lackluster returns and as institutional clients gain more clout.

In the third quarter of 2016, global hedge fund outflows totaled $28 billion — the fourth consecutive quarterly outflow, and the largest since the peak of the financial crisis. This brought outflows for the first nine months of 2016 to $51.5 billion, according to Hedge Fund Research, a Chicago-based industry-tracking firm. Preqin, a London-based alternative assets research firm, put the figure for the same period at $67 billion.

This comes at a time of underperformance for hedge funds, investment vehicles that pool investor capital and employ often risky strategies to generate returns. By one measure, the HFRI Fund Weighted Composite Index, funds worldwide achieved 5.54% in gains during 2016, considerably lagging stock-market benchmarks: the U.K.'s FTSE 100 gained 14.4% in that time, and the U.S. S&P 500 increased by 9.5%.

Hedge funds are increasingly moving away from the traditional "2 and 20" fee structure whereby investors pay 2% of their invested capital in a management fee, together with 20% of returns. Some two-thirds of funds worldwide have abandoned this model, Ermanno dal Pont, Barclays' global head of strategic consultancy, told S&P Global Market Intelligence. Overall, fees for funds launched in 2016 fell to 1.49% and 17.5%, according to HFR.

Changes in fees are most noticeable in classical hedge funds, says dal Pont, but also evident in those specializing in algorithmic strategies. Investors take issue when technology allows firms to drastically reduce their costs.

"When you can run hundreds [of millions], perhaps billions, with three or four people and a server farm doing a lot of the work for you, investors have pushed back on the fees," said Joe Vittoria, CEO of Mirabella, a London-based regulatory hosting platform that works with hedge funds.

A changing investor mix

The pushback over fees partly reflects the changing makeup of hedge fund investors in the period since the 2008 financial crisis.

High-net-worth individuals and family offices now play a smaller role, with many having shifted to better-performing alternative asset classes such as private equity. In 2016, for instance, family offices globally cut their average hedge fund holdings to 8.1% from 9%, and increased their private equity exposure by 2.3% to 22.1%, according to a UBS and Campden Wealth study of 242 such entities.

The large institutional investors that remain, such as pension funds and sovereign wealth funds, are in a position to demand lower fees — particularly since they, too, are threatening to cut their investments. Some 69% of institutional investors plan to decrease their allocation to hedge funds in the next three years, versus 18% planning to increase it, according to EY's 2016 Global Hedge Fund and Investor Survey. Many plan to invest in private equity and real estate.

The pressure has forced contraction in the industry, with some 782 hedge funds liquidated between January and October 2016 — the highest pace since the financial crisis. New launches globally declined to 576, from 785 in the same period of 2015. Large hedge funds have also been slashing their staff numbers: One of Europe's largest funds, Brevan Howard, cut its numbers by 32% last year.

Asset outflows have not been consistent across the board. The biggest hedge funds have seen the most dramatic outflows, while smaller, newer ones have experienced lower outflows or even inflows. In the first nine months of 2016, redemptions at those with more than $5 billion under management totaled $22 billion, HFR research shows, compared to $7.4 billion for firms managing between $1 billion and $5 billion. Funds with less than $1 billion under management enjoyed $1.4 billion of inflows.

"As funds are smaller, they're more nimble — it's borne out in the statistics that the first three years of a fund tend to be the best three years' performance of that investor," Vittoria told S&P Global Market Intelligence. Among the larger funds, he said, there are a lot of strategies that are very crowded, with too many hedge funds chasing too few ideas, and a lack of diversification within the group.

Better times to come?

Hedge funds come into their own in uncertain macroenvironments, though, and fund managers and investors appear bullish about market and geopolitical turbulence lending a helping hand in 2017.

In November 2016, 25% of a group of U.K. hedge fund managers and 108 investors surveyed by Preqin believed Brexit would have a positive effect on their portfolio performance in the next 12 months, compared with only 7% who thought its effect would be negative.

"Very few hedge funds will get every single bet right in 2017, but I would be very disappointed in our industry if we don't see average returns quite higher than they have been," said Vittoria. "If we don't, it tells you there are quite a lot of mediocre managers out there."