Image source: DonkeyHotey on Flickr Creative Commons

Any investor who was wise enough to buy shares of Berkshire Hathaway (BRK.A -0.34%) (BRK.B -0.01%) when Warren Buffett took it over in 1965 has enjoyed one heck of a 50-year run. Berkshire's stock has compounded at a 20.8% annualized rate as of the end of 2015, which blows past the 9.7% return generated by the S&P 500 over the same time period. That phenomenal result is mostly owed to Buffett's knack for buying great businesses at attractive prices, which is why many consider him to be one of the best investors of all time. 

Of course, even the greatest of stock picks does not always go up in a straight line, and history has shown that some of Buffett's holdings can badly trail the market for years on end. With that in mind, here's a closer look at two of his top holdings that have put up poor performances year-to-date. 

Goldman Sachs: Down 9.96%

It has been a rough couple of months to be a shareholder in the investment banking giant. Goldman Sachs' (GS 1.59%) stock has spent all of 2016 trading below its Jan. 1 opening price.

GS Chart

GS data by YCharts

It is not hard to understand why investors haven't been excited about owning the company's shares. Last quarter, Goldman's revenue declined by 13% year-over-year as demand for its investment banking services has waned. In addition, Goldman stated that its investment banking backlog has fallen, which hints that this weakness will continue for the foreseeable future.

Another reason for the soft results was the recent Brexit vote. Roughly 20% of Goldman's revenue comes from its U.K. business, so uncertainty there and in the E.U. more broadly is slowing demand for investment banking services.

Nonetheless, Goldman Sachs continues to muddle through these tough times by controlling its expenses. That's how the company was able to grow its EPS by 88% last quarter even with the headwind of lower revenue. The company also shared the good news that it passed the Federal Reserve's stress test, which gives it the green light to move forward with a capital-return plan that includes buying back stock and raising its dividend. Those moves should reward patient long-term shareholders like Buffett.

Goldman's tangible book value currently stands around $166.90 per share, which is slightly higher than its current share price. That's a historically low valuation for this high-quality business, which suggests that it is a great time for the company to get aggressive with its share repurchase activity.

Wells Fargo: Down 19%

If you've been paying attention to the news then you are likely aware that Wells Fargo (WFC -0.26%) is currently in some hot water. A few weeks back the banking giant admitted that its employees had fraudulently opened as many as 2 million checking and credit card accounts without its customers' permission between 2011 and 2015. Regulators have so far slapped the bank with $185 million in fines, and its CEO was called to testify before Congress. 

Wells Fargo is one of the largest holdings in Berkshire's portfolio, so there's no doubt that Buffett is not happy about this news. The primary worry is that the revelations about the company's malfeasance will cause customers to leave in droves, and hinder the bank's ability to cross-sell financial products in the future. As a result, investors have been dumping shares, and they now trade for less than 1.25 times book value.

WFC Price to Book Value Chart

WFC Price to Book Value data by YCharts

We don't yet know if the news has caused Buffett to reduce Berkshire's massive position in Wells Fargo, but I'd  personally be quite surprised if he did. After all, he is well known for taking the long view on companies that he believes have durable competitive advantages, and it is still unclear if the bank's earnings power will be hindered moving forward.

Is it time to buy Buffett's biggest losers?

With Wells Fargo and Goldman Sachs down big this year, I'm much more inclined to call both stocks buys rather than to suggest that you abandon ship. After all, both of these businesses are hugely important to the global financial system, so there's no doubt that they both have long-term staying power.

With shares of each trading for historically valuations, I think investors could do worse than joining Buffett as a shareholder in either of these banks.