Altria Group (NYSE:MO) may get all the attention, but it is not the only tobacco company making lots of money in the lucrative cigarette industry. Vector Group (NYSE:VGR) is a lesser-known holding company that is primarily composed of its tobacco subsidiary, Liggett Group.

A distant fourth in a market dominated by Altria, Liggett has a special attribute that gives it an important advantage over its larger rival. It also sports one of the highest dividends in the market. As a result, many investors are taking a closer look at this special company. However, it may not be as special as some investors think.

Discount cigarettes
Liggett accounts for 96% of Vector's revenue and 88% of its operating income before corporate overhead. All of Liggett's revenue is derived from branded discount cigarettes. Discount cigarettes is the one segment of the cigarette market that is actually growing. In 2013, Altria's Marlboro volume declined 4.3% and its other premium brand volume declined 10.5%. However, its discount brand volume increased 3.1%. As states raise excise taxes to ever-higher levels, more and more low-income smokers are being priced out of the premium brands. As a result, discount cigarette volume will likely continue to increase.

Discount cigarettes' growth has attracted a number of competitors to the market, causing Liggett to lose ground. In 2013, Liggett captured a 3.3% share of the U.S. discount cigarette market, down from a 3.8% share just two years earlier. On the other hand, Altria's discount cigarette share grew from 3.6% in 2011 to 3.8% in 2013. With Altria's vast distribution network and economies of scale in production, one would expect its share to continue to grow as Liggett's shrinks.

Liggett's special cost advantage
However, Liggett maintains a special cost advantage that may enable it to fend off competitive threats posed by larger rivals. Under the 1998 settlement agreement reached between 46 states and the tobacco industry, cigarette manufacturers must make annual payments to the states in proportion to their market share. However, the first 1.63% of market share is exempt from the settlement. Since Liggett's total share is a little more than 3%, about half of its volume is exempt.

Since people who buy discount cigarettes are primarily concerned about price, Liggett's cost advantage gives it an advantage over larger competitors like Altria. However, it also faces competition from smaller deep-discount manufacturers that also benefit from reduced settlement payments. As a result, the company does not have an advantage over all players in each of its markets.

This may be why Liggett's volume is declining despite its cost advantage over Altria. Last year, Liggett shipped 1.9 billion fewer cigarettes than in 2011. Its smaller rivals' cost advantage makes it difficult for Liggett to grow share, since half of its volume is subject to settlement payments. Liggett does not have the vast distribution capability that enables Altria to grow its discount share, so it is unlikely to outperform either Altria or its smaller competitors in the years ahead. As a result, its dividend may be at risk.

Huge dividend could shrink
It is hard to write about Vector without mentioning its massive dividend. Vector's 7.5% yield is a big draw for dividend investors. However, there are indications that Vector's dividend is not sustainable.

For instance, Vector pays a much higher dividend than it generates in free cash flow. Historically, it financed the difference by borrowing money.

Data source: Morningstar.

However, its $692 million in debt is 5.6 times higher than the $125 million it generates in earnings before interest, taxes, depreciation, and amortization, or EBITDA. Vector paid $109 million in cash to cover interest expenses in 2013, or about 88% of EBITDA. With $407 million in cash and investments, Vector can maintain its dividend for only two or three more years before a drastic reduction will be required.

In addition, it is unlikely that Vector's free cash flow will grow substantially in the coming years. Although higher excise taxes will shift many premium cigarette customers to the discount category, price-sensitive consumers may buy fewer packs. Its precarious market position -- squeezed between wide-distribution Altria and low-price smaller discounters -- is another reason to believe Vector's earning power has little upside. As a result, investors should count on a dividend reduction in the coming years.

Foolish takeaway
Vector's special cost advantage turns out to be not so special after all. Although it gives Liggett a cost advantage over Altria, smaller deep-discount manufacturers enjoy an even better advantage. As a result of declining volumes and no spare debt capacity, Vector's dividend may soon be reduced by a significant amount. Therefore, investors should hesitate before loading up on Vector's stock.

Ted Cooper has no position in any stocks mentioned. The Motley Fool has no position in any of the stocks mentioned. Try any of our Foolish newsletter services free for 30 days. We Fools may not all hold the same opinions, but we all believe that considering a diverse range of insights makes us better investors. The Motley Fool has a disclosure policy.