Do you like the idea of seeing real cash added to your portfolio on a regular basis? Even for growth-minded investors, it can be thrilling to see dividend payments deposited into a brokerage or retirement account every now and then. After all, they're payments that can't be clawed back, and they don't require the market-wide tailwinds typically needed to drive non-dividend-paying stocks higher.

Popular dividend-paying stocks like Chevron and AT&T (T 1.45%) aren't necessarily all they're often made out to be, however. There are plenty of good reasons you wouldn't want to own them. Here's a closer look at four big reasons to avoid dividend-paying stocks.

1. Dividend stocks can create taxable income

Just like the wages employees earn when they're working, dividend payments may be taxable income. Indeed, in many cases -- with dividends categorized as "ordinary" dividends -- they're taxed just like income.

There are a couple of exceptions to this taxability. One of them is "qualified" dividends, which are dividend payments that meet certain criteria laid out by the IRS. These payments are generally taxed at a lower rate than income, and in some circumstances aren't taxed at all. The other exception is dividends paid by stocks or funds held within a tax-deferred vehicle like an IRA (individual retirement account). These aren't taxed as they're paid either.

If neither of these exceptions applies to you and you're looking to keep your current tax bills to a minimum, dividend stocks may not be ideal for you.

2. Dividends tend to be associated with lower-growth companies

Don't misunderstand. There are high-growth companies that also pay respectable dividends. They're the exception rather than the norm, however. In general, corporations that pay out a sizable portion of their profits as dividends tend to be cash distribution vehicles for slower-growth businesses like banking, consumer goods, and utilities. Technology stocks, cyclical stocks, and certain other industries tend to keep the bulk of their profits rather than share them with shareholders, ultimately investing those earnings back into related but new opportunities.

3. High and rising interest rates work against their prices

It's not been a factor in years, since interest rates have been at rock-bottom levels since 2009. Since early last year, however, the Fed Funds rate has soared to multi-year highs, and it is still inching higher.

That matters for one simple reason. The market adjusts to higher interest rates by lowering the prices of dividend-paying stocks in order to better align their dividend yields to prevailing rates. These adjusted prices still reflect factors like risk and growth prospects, but when investors have other income options like newly higher-yielding bonds, something of a price concession gets built into dividend-paying stocks.

4. The dividend can become more important than the business itself

Kudos to companies that recognize and respect that many of their shareholders may own their stock predominantly for the dividend. These organizations often strive to continue making -- and even growing -- their payouts in order to maintain their stocks' appeal, even when it's fiscally difficult to do so.

It's not unheard of, however, for a dividend-paying company to become more concerned about maintaining a dividend pedigree than running a sustainable business. The irony? Such a mindset can ultimately make it even more difficult to continue funding a dividend payment.

The aforementioned AT&T comes to mind. While it was an income-driving champ for years, as telecom saturation neared and the company used debt to buy its way into the media and entertainment industry, paying the dividend became increasingly problematic. The stock's performance over the past few years speaks for itself. 

Just think it through

None of this is meant to steer you away from any and all dividend-paying stocks under any and all circumstances. There are good reasons to own dividend stocks, and there are plenty of great companies out there making sustainable dividend payments.

Rather, it's simply to say you should think carefully about stepping into a dividend-paying name. Does it actually help you meet your long-term investing goal, and can it continue doing so for the foreseeable future? Is your initial principal in jeopardy in exchange for plugging into what looks like a healthy income stream? There are always trade-offs. Your job as an income-minded investor is minimizing your prospective risk and downside of a dividend stock, and maximizing your particular upside.