Many income investors are drawn to stock with high yields. But as I discussed in a recent article, the yield should never be prioritized above a company's earnings growth, free cash flow, and payout ratios. If a stock has a high yield but doesn't pass those three tests, the dividend is probably unsustainable.

Image source: Getty Images.

But one odd company seems to break all the rules -- Vector Group (NYSE:VGR), a hybrid tobacco and real estate company that pays a hefty forward yield of 6.8%. At first glance, Vector looks like a dividend trap, since it can't cover its dividend payments with its earnings or free cash flow.

Yet the company has paid uninterrupted cash dividends since 1995, and the stock has rallied more than 80% over the past 10 years. Let's take a closer look at this oddball income generator and whether its dividend can survive over the long term.

What does Vector Group do?

Vector's tobacco business sells Liggett cigarette brands like Pyramid and Eve. Revenue from that unit rose 3.8% annually to $274.2 million, thanks to favorable pricing and higher shipment volumes. The tobacco business generated 88% of Vector's adjusted EBITDA last quarter.

The core of Vector's real estate portfolio is its majority stake in real estate broker Douglas Elliman, but it also includes interests in a wide range of hotels, residential buildings, and commercial properties. That segment's revenue fell 0.4% to $184.9 million last quarter, and it accounted for 17% of Vector's adjusted EBITDA. The combined percentage of the two units exceeds 100% due to losses in the e-cigarettes, corporate, and other segments.

Vector's total revenue rose 2% annually to $459.1 million last quarter, and adjusted EBITDA attributed to the company rose 4% to $75.1 million. Its adjusted net earnings nearly doubled to $0.19 per share. The single analyst who covers Vector expects its revenue to rise 1% this year and for its earnings to jump 35% this year.

Why Vector's dividend is strange

Vector pays both a quarterly cash dividend and a 5% annual stock dividend. It recently declared a quarterly cash dividend of $0.40 per share, which remains unchanged since 2014. That dividend looks unsustainable, since it equals an earnings-based payout ratio of 263%.

It's also unsustainable on a free cash flow basis -- Vector's dividend payments over the past 12 months gobbled up 153% of its FCF. By comparison, Altria (NYSE:MO) -- which has a forward yield of 3.7% -- has an earnings-based payout ratio of 87%, and it spent 88% of its FCF on dividends over the past 12 months.

In addition to that big cash dividend, Vector has paid an annual 5% stock dividend since 1999. This means that Vector "gifts" new shares to existing shareholders based on their existing positions. For example, a shareholder with 100 shares receives five new shares during the first year. Those shares pile up via compounding every year, boosting its quarterly cash dividends.

How does Vector fund its dividend payments?

Vector's big cash dividend and generous stock dividend make it a popular pick among income investors, but there are clear signs that this party can't last forever. Vector can maintain payout ratios above 100% because it partly funds its dividends with debt. This is painfully clear when we look at the growth of its long-term debt over the past decade.

VGR Total Long Term Debt (Quarterly) Chart

Data source: YCharts.

That rising debt load is worrisome since Vector finished last quarter with just $240 million in cash and equivalents. As interest rates rise, taking on more debt to fund, its dividends will worsen the situation. Vector's 5% stock dividend has also gradually increased its outstanding share count over the past decade. That's an odd strategy, since tobacco companies like Altria generally repurchase stock to tighten up their earnings and valuations.

VGR Shares Outstanding Chart

Data source: YCharts.

That increased share count has boosted Vector's trailing P/E to 38 -- which is much higher than Altria's 25 and the industry average of 21 for cigarette companies.

The bottom line

Vector had a great run over the past decade, but it seems like investors have realized that regular tobacco companies like Altria or Reynolds American (NYSE:RAI) are safer long-term income plays. Vector declined 6% this year, while Altria and Reynolds respectively rallied 14% and 21%. With interest rates set to rise, dividend stocks with high P/Es and shaky fundamentals -- like Vector Group -- could be aggressively sold off.

Vector might still be able to pay its dividend for years, but it eventually needs to slash its payout to more reasonable levels. When that happens, investors will quickly flock to its lower-yield peers.

This article represents the opinion of the writer, who may disagree with the “official” recommendation position of a Motley Fool premium advisory service. We’re motley! Questioning an investing thesis -- even one of our own -- helps us all think critically about investing and make decisions that help us become smarter, happier, and richer.