Dividend stocks can be a great way to build wealth over time. Still, there's more to investing than just watching your dividends roll in, and focusing too much on just those payments can trap you in a world of trouble.

As an investor, it's important to find a balance between your dividend stocks and the rest of your portfolio. These four reasons to avoid dividend-paying stocks can help you find that balance while still keeping yourself on track for a brighter long-term financial future.

Person turning down cash in a business setting.

Image source: Getty Images.

No. 1: You're trying to keep your income down

Although the cash from dividends is great to get, if you're receiving dividends in an after-tax account, it does count as income when calculating your Adjusted Gross Income.  Things like Obamacare premium subsidies, Medicare Part B premiums, and your federal income tax brackets are heavily influenced by your adjusted gross income.

If you're looking to keep your total income down to stay on the good side of a tax bracket or a subsidy, then minimizing your exposure to dividend-paying stocks can help you do that. After all, the dividends you aren't receiving will never show up in your adjusted gross income.

No. 2: You need spending cash from your portfolio

It's very tempting to look at dividends as if they were a source of reliable spending cash that you can tap from your portfolio to make ends meet. The challenge with that approach is that dividends are never guaranteed payments. Even worse, if a company is forced to cut its dividend, its stock usually tumbles as well. That can leave you both without a source of income and without enough cash to make up the lost income elsewhere.

Instead of relying directly on your dividends for spending cash, use them as a part of your plan to maintain your investment-grade bond ladder or other higher-certainty form of cash. That way, if your dividends do get cut, you have that buffer that better protects your ability to cover your current spending needs. If nothing else, that should at least buy you more time to build a more robust recovery plan from any unfortunate dividend reductions that may occur.

No. 3: The yield looks too good to be true

The only way a company can sustain its dividend over time is by paying the dividend from its continuing earnings or operating cash flows. If a company's dividend can no longer be supported sustainably, its stock tends to drop a bit in anticipation of a likely future dividend cut. That fall in stock price leads the company to be priced with what looks like a high yield, as that yield is calculated by comparing its most recent or newly announced dividend with its stock price.

In reality, that's only attractive if the company manages to recover to the point where it can support its dividend over time from its continuing operations. If it can't, then the more likely outcome is a dividend cut and further falling stock price as that yield trap is ultimately revealed for what it was.

No. 4: Fast growth potential is more important to you than current income

If there's a structural downside to dividends, it's that every dollar a company pays in its dividend is a dollar it can't invest in growing its business. That tends to mean that companies that don't pay dividends have the opportunity to grow faster than otherwise equivalent businesses that do pay dividends can grow. As a result, if your objective is to maximize the possibility for growth in your portfolio, seeking out companies that actively reinvest every dollar in their growth opportunities may be a better option.

Just be wary that there is such a thing as "empire-building" behavior among some company leaders. Folks acting like that are seeking to grow the business or the organization more for the sake of leading a larger organization rather than to maximize long-term returns. Be sure to check in on the financials and listen to the company's financial calls to make sure the business is focused on the right long-term goals. Otherwise, it's hard to justify a company keeping that cash instead of distributing it to its shareholders.

Invest in healthy companies for the long term

No matter what type of business you're looking to invest in, having an eye toward the long term is an incredibly important part of giving you a great chance for success. Both dividend and non-dividend stocks might play a role in your portfolio, but neither is an appropriate investment for your near-term cash needs.

With a focus on the long-term value of the companies you're investing in and a long-term mindset while you're investing in them, it becomes easier to make smart buy and sell decisions in volatile times. That can help you build a better foundation for the long term wealth you're looking to build, regardless of whether the stocks you buy pay dividends or not.