The premise seems sound enough. Warren Buffett is arguably one of the world's most successful investors, with a track record to back up the claim. His company, Berkshire Hathaway (BRK.A 1.18%) (BRK.B 1.30%), has outperformed the market for the long haul, up nearly 4,000% since 1990 versus the S&P 500's 800% gain for the same timeframe. It's lagged of late, but the Oracle of Omaha and his proteges may just be wisely biding their time.

Except, maybe mirroring the Buffett way isn't exactly the best fit for all investors.

Here are four reasons some people might want to think twice before blindly taking advice from Berkshire's veteran stock-picking guru.

Woman not listening by holdings fingers in her ears

Image source: Getty Images.

1. Sticking to "buying what you know" precludes learning

"Buy what you know" may be one of Warren Buffett's most repeated axioms, and the point is well taken. Unless someone really knows a particular business in and out, it's difficult to judge a related company's prospects.

Sometimes, though, an investor must leave their comfort zone and step into unfamiliar territory, when there's no real opportunity in that individual's area of expertise.

That's not to suggest someone should simply hold their nose, dive into a strange sector, and then hope for the best. Careful study is always a good idea, and investors would be wise to wade slowly into unusual waters. As time marches on, though, an investor who learns about new types of businesses opens up a whole new world of options.

2. Buffett's not investing for retirement, but you (probably) are

As of the latest tally, Forbes estimated Warren Buffett's net worth was somewhere on the order of $70 billion, presumably the bulk of which consists of Berkshire shares. The fund isn't bulletproof; it can suffer when the broad market does. But it almost goes without saying that with access to a stash that's currently worth $70 billion, Buffett will never need to worry about money again. He'll never even need to worry about market downturns again.

Not so for most everyone else. The average investor seeking to build a retirement nest egg will need to work very hard during their earning years to drive growth and then work very hard in their golden years to properly manage withdrawals of those funds. Perhaps most of all, the average investor will have to make sure they're not making the shift from a risk-on to a risk-off mode in the middle of a downturn that Buffett doesn't have to worry about.

3. Sorry, but diversification isn't dangerous

To be fair, the Oracle of Omaha likely says it more for effect than as advice to strictly adhere to. But, his comment "diversification is protection against ignorance" (sometimes voiced as "diversification is dangerous") runs the risk of inadvertently steering someone into trouble.

He's right, to be sure -- diversification is protection against ignorance. Because the typical investor simply doesn't have the time to watch, learn, and know everything, a diversified portfolio provides much-needed protection against a substantial industry downturn. The irony is, Berkshire's lack of diversity is a key reason it's underperformed the broad market since April.

Although Berkshire owns an oversized stake in Apple shares, which have easily led the market of late, heavy exposure to financial stocks like Bank of America and consumer goods names such as Coca-Cola have dragged Berkshire's performance down. While the S&P 500 is up 39% from its mid-March low, financial names have only advanced 30%. Staples stocks are only up about 24% from that low. Even before March though, Berkshire Hathaway's performance had been subpar since early 2019, mostly thanks to a terrible performance from an overweighted position in Kraft Heinz. That's a long time to wait for a non-diversified portfolio to rebound.

Those lagging sectors may eventually close the gap, but they may not do so anytime soon.

4. Berkshire's public stocks aren't the entire portfolio

Finally, while Berkshire's long-term track record may speak for itself despite its recent struggle, there's an oft-overlooked secret about the fund: It's not just stocks. It holds more than $200 billion worth of publicly traded equities like the aforementioned Apple and Bank of America, but there's around that much again worth of privately held enterprises such as GEICO Insurance, McLane Company, and railroad name Burlington Northern Santa Fe.

It's an important distinction for a couple of reasons. One of them is, whereas changes in the prices of stocks in the Berkshire portfolio help or hurt the net asset value of a share of Berkshire, these private holdings are ultimately valued based on the cash flow they contribute to the pool of investments. The second important distinction is, the income orientation of these privately held enterprises means the fund's managers may pick stocks differently for the publicly traded piece of the portfolio than most ordinary investors might.

Bottom line

Buffett's trades are certainly worth monitoring as idea generators. His timeframe is likely to be very different than yours, though, and he can stomach a little more risk with his stock picks because Berkshire's got a huge non-stock, cash-generating collection of companies to smooth out the rough patches. You probably don't have the same luxury.