Buying stocks with big dividend yields can be a smart way to generate a stable flow of passive income. But if a stock's yield looks too good to be true, it might be unsustainable. One way to get a bead on whether a company's dividend is sustainable is to check its payout ratio, which can be calculated in two ways.

Source: Pixabay.

The first way is to calculate the percentage of a company's earnings per share paid out as dividends over the past four quarters. But since earnings can be inflated by buybacks and adjustments, many investors prefer using free cash flow (FCF) to gauge a company's ability to keep paying its dividend.

Generally speaking, weak earnings and FCF growth and payout ratios exceeding 100% indicate that a company's dividend could be slashed in the near future. Let's take a look at two companies which fit that profile -- Vector Group (VGR -0.10%) and Frontier Communications (FTR).

Vector Group
Vector Group owns a portfolio of domestic tobacco brands and real estate holdings. Last quarter, 63% of its revenue came from its domestic tobacco business, which sells Liggett cigarette brands like Pyramid and Eve. The rest came from its real estate business, which includes hotels, residential buildings, and commercial properties.

During the quarter, Vector's GAAP revenue rose 3% annually to $430.3 million, with tobacco sales falling 1% and real estate revenue rising 3%. Net income plummeted 35% to just $7.9 million.

Yet over the past 12 months, Vector paid out 143% of its free cash flow as dividends, and it currently pays a hefty forward yield of 7%.

Vector's sliding earnings, triple-digit payout ratio, and huge dividend yield all suggest that its dividend could be slashed soon. But during last quarter's conference call (as transcribed by Seeking Alpha), CEO Howard Lorber declared that Vector's dividend policy "remains the same", and that the company has had an "uninterrupted track record of regular, quarterly cash dividends since 1995 and an annual 5% stock dividend since 1999." Analysts expect Vector's annual earnings to grow 11% over the next five years, which might prop up its dividend, but I still wouldn't buy this stock as a long-term income generator.

Vector's cigarette brands. Image Source: Vector Group.

Frontier Communications
Telecom company Frontier Communications provides services to rural areas and smaller cities across the U.S. It recently completed the purchase of Verizon's (VZ -0.53%) wireline assets in California, Texas, and Florida for $10.5 billion. It also bought AT&T's (T 1.88%) wireline business in Connecticut.

Last quarter, Frontier's revenue rose 6% to $1.41 billion, but much of that gain came from inorganic growth instead of its core voice and Internet businesses. Non-GAAP net income rose 60% annually to $56 million, but its GAAP-adjusted net loss widened from $14 million a year earlier to $103 million due to its aforementioned acquisitions. Looking ahead, Frontier expects to generate "day one" synergies of $600 million from the Verizon deal -- indicating that losses could eventually narrow over the long term -- but analysts don't expect it turn return to GAAP-adjusted profitability anytime soon.

Over the past 12 months, Frontier paid out a whopping 132% of its free cash flow as dividends, and currently pays a forward yield of 7.5%. Last quarter, Frontier claimed that "excluding the additional dividends paid on the common and preferred stock issued in the June 2015 equity offerings and interest costs on our September 2015 private notes offering," the dividend accounted for a "sustainable" 43% of the company's free cash flow. However, excluding those dividends doesn't make much sense, since Frontier is still obligated to pay those new shareholders dividends in the future. Frontier also has a history of reducing its dividend, so investors shouldn't be surprised if it slashes its payout again.

Find the better alternatives
Tobacco and telecom stocks often pay some of the highest dividends in the market, but Vector and Frontier look more like high-yield traps than sustainable income stocks. 

Tobacco investors can consider buying market leader Altria (MO 0.12%) instead, which spent a more sustainable 75% of its free cash flow on dividends over the past 12 months. Telco investors can consider investing in AT&T -- which paid out 64% of its free cash flow as dividends over the past year -- instead of Frontier. Altria's and AT&T's forward yields of 3.6% and 4.9% don't look as "generous" as Vector's and Frontier's yields, but they certainly look much more sustainable over the long term.