Last week I took a peek under the hood of Warren Buffett's baby, Berkshire Hathaway (NYSE: BRK-A)(NYSE: BRK-B), and offered three solid reasons that the conglomerate is a worthwhile buy right now.

As any Foolish investor knows, though, there's no stock or company without its warts, and Berkshire Hathaway is no exception. So with that in mind, let's take a look at three compelling reasons not to buy Berkshire -- or even, dare I say, sell it if you're a current owner.

1. Opportunity elsewhere
There are a lot of great things that I can say about Berkshire Hathaway, but one thing that I'd have trouble saying is that it's going to deliver exceptional growth in the years ahead. A lot of this simply has to do with Berkshire's size. As a company worth more than $200 billion with annual revenue of nearly $150 billion, Berkshire can certainly grow further, but it's much tougher for a company that size to double than one that's a fraction of that size.

This is hardly a dire prediction for Berkshire -- 10 years ago, this was still a very large company, and yet net income has grown at an average annual rate of nearly 11% during that period. Revenue has more than tripled over that same decade. And if the economy manages to find its way back to a growth path, that would be a great tailwind for Berkshire.

However, there are plenty of companies that are growing much faster than Berkshire. Over the past five years, Berkshire's revenue growth has averaged 6%. Of the S&P 500 companies, 211 have had faster growth than that. That group includes Apple (Nasdaq: AAPL) -- proving with its 46% annual revenue growth that big companies can still grow fast -- and Intuitive Surgical (Nasdaq: ISRG) -- whose next-generation surgical tools led to 34% annual revenue growth.

In short, for investors looking for high growth and the potential for a big homerun, Berkshire may not be the place to be.

2. No more Buffett
The upshot to the concerns over Buffett's eventual exit from Berkshire is that it's keeping the stock's valuation down, making it a more attractive buy. However, there will be a very real impact on Berkshire when Warren Buffett is no longer at the helm.

To understand why, we need to understand exactly how Buffett fits into the Berkshire machine. What Buffett isn't doing is managing the day-to-day operations of the myriad companies under the conglomerate's umbrella. He's not telling GEICO how to spend its marketing dollars any more than he's telling Brooks what colors it should offer its popular Ghost running shoes in.

What Buffett does do is take the company's excess profit -- that is, profit that can't be reinvested in the individual Berkshire businesses for an acceptable rate of return -- and put that to work elsewhere. That might mean buying a company in whole, as it did with BNSF Railway, or it might mean taking a partial stake in a public company, as it did with its eyebrow-raising investment in IBM (NYSE: IBM). When other opportunities arise, such as buying preferred stock in Goldman Sachs as Berkshire did during the financial crisis, that's yet another potential parking spot for Berkshire's funds.

Buffett's impressive prowess in allocating Berkshire's capital has been a key source of the company's success. Without Buffett, the profits will still be there, but the savvy reallocation may not be.

3. A better day to buy
Although the inevitable departure of Buffett already seems to be putting pressure on the stock, it's a near certainty that on the day that Buffett decides to step down (or passes on), there will be an impact on the stock. The stock may not recover quickly and could face even more pressure in the months that follow as the market struggles with digesting Berkshire sans Buffett.

I happen to think that Berkshire will still be a great company even after it's no longer headed by Buffett. But depending on the timing and circumstances of his departure and the extent to which Berkshire has a solid succession plan in place, investors buying today could see their holdings take a hit. On the other hand, those who wait could find themselves with an even better buying opportunity when the market (over)reacts to Buffett's departure.

I'm not selling
I've rated Berkshire Hathaway an outperformer in my Motley Fool CAPS portfolio and I own it, and plan to continue to own it, in my personal portfolio. But I'm under no illusions that this is a perfect investment. For the reasons above -- and there are other risks as well -- Berkshire may not be right for some investors.

Those who do skip Berkshire in hopes of finding a higher-growth buy may find Apple on their short list of potential buys. But is Apple really a better bet? Senior analyst Eric Bleeker has tackled exactly that question in the new premium report "Can Apple Still Juice Your Portfolio?"

The Motley Fool owns shares of Berkshire Hathaway, International Business Machines, Apple, and Intuitive Surgical. Motley Fool newsletter services have recommended buying shares of Berkshire Hathaway, Goldman Sachs Group, Apple, and Intuitive Surgical. Motley Fool newsletter services have recommended creating a bull call spread position in Apple and a synthetic long position in International Business Machines. We Fools may not all hold the same opinions, but we all believe that considering a diverse range of insights makes us better investors. Try any of our Foolish newsletter services free for 30 days.

Fool contributor Matt Koppenheffer owns shares of Berkshire Hathaway, but does not have a financial interest in any of the other companies mentioned. You can check out what Matt is keeping an eye on by visiting his CAPS portfolio, or you can follow Matt on Twitter @KoppTheFool or Facebook. The Fool's disclosure policy prefers dividends over a sharp stick in the eye.